Dynamic programming
It is used, when the solution can be
recursively described in terms of solutions
Dynamic programming
to subproblems (optimal substructure)
Longest Common Subsequence
Algorithm finds solutions to subproblems
and stores them in memory for later use
More efficient than “brute-force methods”,
which solve the same subproblems over
and over again
1 2
Longest Common Subsequence LCS Algorithm
(LCS) if |X| = m, |Y| = n, then there are 2m
subsequences of x; we must compare each
Application: comparison of two DNA strings with Y (n comparisons)
Ex: X= {A B C B D A B }, Y= {B D C A B A} So the running time of the brute-force
Longest Common Subsequence: algorithm is O(n 2m)
X= AB C BDAB Notice that the LCS problem has optimal
substructure: solutions of subproblems are
Y= BDCAB A parts of the final solution.
Subproblems: “find LCS of pairs of prefixes
of X and Y”
3 4
1
Longest Common Subsequence Example
Definition 1: Subsequence
Given a sequence X = <A,B,D,F,M,Q>
X = < x1, x2, . . . , xm> Z = <B, F, M>
then another sequence
Z = < z1, z2, . . . , zk>
is a subsequence of X if there exists a strictly Z is a subsequence of X with index
increasing sequence <i1, i2, . . . , ik> of indices of x sequence <2,4,5>
such that for all j = 1,2,...k we have xij= zj
5 6
More Definitions
Example
Definition 2: Common subsequence
– Given 2 sequences X and Y, we say Z is a X = <A,B,C,B,D,A,B>
common subsequence of X and Y if Z is a
Y = <B,D,C,A,B,A>
subsequence of X and a subsequence of Y
Then, what is/are LCS?
Definition 3: Longest common
1. BCBA length 4
subsequence problem
– Given X = < x1, x2, . . . , xm> and Y = < y1,
y2, . . . , yn> find a maximum length
2. BDAB length 4
common subsequence of X and Y
3. BCAB length 4
7 8
2
Brute Force Algorithm
Brute force algorithm would compare each subsequence of
X with the symbols in Y Yet More Definitions
1 for every subsequence of X
2 Is there subsequence in Y?
Definition 4: Prefix of a subsequence
If X = < x1, x2, . . . , xm> , the ith prefix of X for
3 If yes, is it longer than the longest i = 0,1,...,m is Xi = < x1, x2, . . . , xi>
subsequence found so far?
What about Complexity? Example
if|X| = m, |Y| = n, then there are 2m subsequences – if X = <A,B,C,D,E,F,H,I,J,L> then
of x; we must compare each with Y (n comparisons)
X4 = <A,B,C,D> and X0 = <>
So the running time of the brute-force algorithm is
O(n 2m)
9 10
Optimal Substructure Sub-problem structure
Theorem Optimal Substructure of LCS Case 1
Let X = < x1, x2, . . . , xm> and Y = < y1, y2, . . . , yn> – if xm = yn then there is one sub-problem to solve
be sequences and let Z = < z1, z2, . . . ,zk> be any find a LCS of Xm-1 and Yn-1 and append xm
LCS of X and Y
1. if xm = yn then zk = xm = yn and zk-1 is an LCS of
Case 2
xm-1 and yn-1 – if xm yn then there are two sub-problems
2. if xm yn and zk x m Z is an LCS of Xm-1 and Y • find an LCS of Xmand Yn-1
3. if xm yn and zk yn Z is an LCS of Xm and Yn-1 • find an LCS of Xm-1 and Yn
• pick the longer of the two
11 12
3
Cost of Optimal Solution The Recurrence
Cost is length of the common 0 if i 0 or j 0
subsequence
c[i, j] c[i 1, j 1] 1 if i, j 0 and x i y j
We want to pick the longest one
max(c[i, j 1],c[i 1, j]) if i, j 0 and x i y j
Let c[i,j] be the length of an LCS of the
sequences Xi and Yj
Base case is an empty subsequence--
then c[i,j] = 0 because there is no LCS
13 14
Tables used for LCS LCS recursive solution
c[i 1, j 1] 1 if x[i ] y[ j ],
c[i, j ]
max(c[i, j 1], c[i 1, j ]) otherwise
Table c[0..m, 0..n] stores the length of
an LCS of Xi and Yj c[i,j] We start with i = j = 0 (empty substrings of x
Table b[1..m, 1..n] stores pointers to and y)
optimal sub-problem solutions Since X0 and Y0 are empty strings, their LCS
is always empty (i.e. c[0,0] = 0)
LCS of empty string and any other string is
empty, so for every i and j: c[0, j] = c[i,0] = 0
15 16
4
LCS recursive solution LCS recursive solution
c[i 1, j 1] 1 if x[i ] y[ j ],
c[i 1, j 1] 1 if x[i ] y[ j ], c[i, j ]
c[i, j ] max(c[i, j 1], c[i 1, j ]) otherwise
max(c[i, j 1], c[i 1, j ]) otherwise
When we calculate c[i,j], we consider two Second case: x[i] != y[j]
cases: As symbols don’t match, our solution is not
First case: x[i]=y[j]: one more symbol in improved, and the length of LCS(Xi , Yj) is
strings X and Y matches, so the length of LCS the same as before (i.e. maximum of LCS(Xi,
Xi and Yj equals to the length of LCS of Yj-1) and LCS(Xi-1,Yj)
smaller strings Xi-1 and Yi-1 , plus 1 17 18
LCS Length Algorithm LCS Example
LCS-LENGTH(X,Y)
1 m length[X]
We’ll see how LCS algorithm works on the
2 n length[Y] following example:
3 for i to m
4 do c[i,0] X = ABCB
5 for j to n
6 do c[0,j] Y = BDCAB
7 for i to m
8 do for j to n
9 do if xi = yj
then c[i,j] c[i-1,j-1] + 1
What is the Longest Common Subsequence
b[i,j] ”” of X and Y?
12 else if c[i-1,j] c[i,j-1]
13 then c[i,j] c[i-1,j]
14 b[i,j] ”” LCS(X, Y) = BCB
15 else c[i,j] c[i,j-1]
16 b[i,j] ”” X=AB C B
17 return c and b 19
Y= BD CAB 20
5
ABCB ABCB
LCS Example (0) BDCAB
LCS Example (1) BDCAB
j 0 1 2 3 4 5 j 0 1 2 3 4 5
i Yj B D C A B i Yj B D C A B
0 Xi 0 Xi 0 0 0 0 0 0
1 A 1 A 0
2 B 2 B 0
3 C 3 C 0
4 B 4 B 0
X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5 for i = 1 to m c[i,0] = 0
Allocate array c[5,4] for j = 1 to n c[0,j] = 0
21 22
ABCB
LCS Example (15) BDCAB LCS Algorithm Running Time
j 0 1 2 3 4 5
i Yj B D C A B
0 Xi 0 0 0 0 0 0 LCS algorithm calculates the values of each
1 A 0 0 0 0 1 1
entry of the array c[m,n]
B So what is the running time?
2 0 1 1 1 1 2
3 C 0 1 1 2 2 2 O(m*n)
4 B 0 1 1 2 2 3 since each c[i,j] is calculated in
if ( Xi == Yj ) constant time, and there are m*n
c[i,j] = c[i-1,j-1] + 1 elements in the array
else c[i,j] = max( c[i-1,j], c[i,j-1] )
35 36
6
How to find actual LCS How to find actual LCS - continued
So far, we have just found the length of LCS, Remember that
but not LCS itself. c[i 1, j 1] 1 if x[i ] y[ j ],
c[i, j ]
We want to modify this algorithm to make it
max(c[i, j 1], c[i 1, j ]) otherwise
output Longest Common Subsequence of X
and Y So we can start from c[m,n] and go backwards
Each c[i,j] depends on c[i-1,j] and c[i,j-1] Whenever c[i,j] = c[i-1, j-1]+1, remember
or c[i-1, j-1] x[i] (because x[i] is a part of LCS)
For each c[i,j] we can say how it was acquired: When i=0 or j=0 (i.e. we reached the
2 2 For example, here beginning), output remembered letters in
c[i,j] = c[i-1,j-1] +1 = 2+1=3 reverse order
2 3 37 38
Algorithm to print LCS
Finding LCS
j 0 1 2 3 4 5
i Yj B D C A B
PRINT-LCS (b,X,i,j) 0 Xi 0 0 0 0 0 0
1if i = 0 or j = 0 A
2 then return 1 0 0 0 0 1 1
3 if b[i,j] = ”” 2 B 0 1 1 1 1 2
4 then PRINT-LCS(b,X,i-1,j-1)
5 print xi 3 C 0 1 1 2 2 2
6 else if b[i,j] ”” 4 B 0 1 1 2 2 3
7 then PRINT-LCS(b,X,i-1,j)
8 else PRINT-LCS(b,X,i,j-1)
39 40
7
Finding LCS (2)
j 0 1 2 3 4 5
i Yj B D C A B
0 Xi 0 0 0 0 0 0
1 A 0 0 0 0 1 1
2 B 0 1 1 1 1 2
3 C 0 1 1 2 2 2
4 B 0 1 1 2 2 3
LCS (reversed order): B C B
LCS (straight order): B C B
(this string turned out to be a palindrome) 41
