61A Lecture 35
Monday, 28th November, 2011
Monday, November 28, 2011
Last time: sequential data and iterators
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
 can’t deal with infinite sequences
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
 can’t deal with infinite sequences
Iterators
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
 can’t deal with infinite sequences
Iterators
 Store how to compute elements
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
 can’t deal with infinite sequences
Iterators
 Store how to compute elements
 Compute one element at a time
2
Monday, November 28, 2011
Last time: sequential data and iterators
Sequences
 The sequence abstraction so far
 Length
 Element selection
• Lists and tuples
 Store all elements up-front
 can’t deal with huge data
 can’t deal with infinite sequences
Iterators
 Store how to compute elements
 Compute one element at a time
 Delay evaluation
2
Monday, November 28, 2011
Last time: sequential data and iterators
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
 for-loops rely on these methods
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
 for-loops rely on these methods
Generator functions
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
 for-loops rely on these methods
Generator functions
 Functions that use yield to output values
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
 for-loops rely on these methods
Generator functions
 Functions that use yield to output values
 Creates a generator object
3
Monday, November 28, 2011
Last time: sequential data and iterators
Streams -- a unit of delayed evaluation.
 2 elements, first and rest.
 “first” is stored
 “compute_rest” is stored
 calculate “rest” on demand
Native python iterator interface
 __iter__()
 __next__()
 for-loops rely on these methods
Generator functions
 Functions that use yield to output values
 Creates a generator object
 __iter__() and __next__() automatically defined
3
Monday, November 28, 2011
Today: modularity, processing pipelines, and
coroutines
Modularity in programs so far
 Helper functions a.k.a “subroutines”
Coroutines: what are they?
Coroutines in python
Types of coroutines
Multitasking
4
Monday, November 28, 2011
Modularity so far: helper functions
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
 A sub-program responsible for a
small piece of computation
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
Main function
• a.k.a. “subroutines”
 A sub-program responsible for a
small piece of computation
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
subroutine
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
subroutine
subroutine
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
subroutine
subroutine
subroutine
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
subroutine
subroutine
subroutine
subroutine
5
Monday, November 28, 2011
Modularity so far: helper functions
Modularity in programming?
 Helper functions!
• a.k.a. “subroutines”
Main function
subroutine
 A sub-program responsible for a
small piece of computation
subroutine
subroutine
A main function is
responsible for calling all
the subroutines
subroutine
subroutine
5
Monday, November 28, 2011
Modularity with Coroutines
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
coroutine
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
6
Monday, November 28, 2011
Modularity with Coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
Coroutines are also sub-computations
The difference: no main function
Separate coroutines link together to form a complete pipeline
6
Monday, November 28, 2011
Coroutines vs. subroutines: a conceptual difference
coroutine
Main function
subroutine
coroutine
coroutine
coroutine
subroutine
coroutine
subroutine
subroutine
subroutine
7
Monday, November 28, 2011
Coroutines vs. subroutines: a conceptual difference
coroutine
Main function
subroutine
coroutine
coroutine
coroutine
subroutine
coroutine
subroutine
subroutine
subroutine
subordinate to a main function
7
Monday, November 28, 2011
Coroutines vs. subroutines: a conceptual difference
coroutine
Main function
subroutine
coroutine
coroutine
coroutine
subroutine
coroutine
subroutine
colleagues that cooperate
subroutine
subroutine
subordinate to a main function
7
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
current = 'a'
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
current = 'a'
while current <= 'd':
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
current = 'a'
while current <= 'd':
yield current
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
pauses execution
 Consume data with yield
value = (yield)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
pauses execution
local variables preserved
value = (yield)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
pauses execution
local variables preserved
resumes when .send(data) is called
value = (yield)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
send(data)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
value = (yield)
send(data)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
value = (yield)
send(data)
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
value = (yield)
send(data)
(yield) returns the sent data.
8
Monday, November 28, 2011
Coroutines in python, or, the many faces of “yield”
Previously: generator functions
 Produce data with yield
pauses execution
local variables preserved
resumes when .__next__ is called
returns the yielded value
def letters_generator():
current = 'a'
while current <= 'd':
yield current
current = chr(ord(current)+1)
Now: coroutines
 Consume data with yield
value = (yield)
pauses execution
local variables preserved
resumes when .send(data) is called
assigns value to yielded data
value = (yield)
send(data)
(yield) returns the sent data.
Execution resumes
8
Monday, November 28, 2011
Coroutines in Python
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
 Execution pauses waiting for data to be sent
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
 Execution pauses waiting for data to be sent
Send a coroutine data using send(...)
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
 Execution pauses waiting for data to be sent
Send a coroutine data using send(...)
Start a coroutine using ___next__()
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
 Execution pauses waiting for data to be sent
Send a coroutine data using send(...)
Start a coroutine using ___next__()
Signal the end of a computation using close()
9
Monday, November 28, 2011
Coroutines in Python
Consuming data with yield:
 value = (yield)
 Execution pauses waiting for data to be sent
Send a coroutine data using send(...)
Start a coroutine using ___next__()
Signal the end of a computation using close()
• Raises GeneratorExit exception inside coroutine
9
Monday, November 28, 2011
Example: print out strings that match a pattern
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
execution starts
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
execution starts
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
stops here, waiting
s = (yield)
for data
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
stops here, waiting
s = (yield)
for data
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
stops here, waiting
s = (yield)
for data
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
execution starts
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
‘the Jabberwock with eyes of flame’
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
‘the Jabberwock with eyes of flame’
Step 4: close the coroutine
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
‘the Jabberwock with eyes of flame’
Step 4: close the coroutine
>>> m.close()
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
catch exception
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
‘the Jabberwock with eyes of flame’
Step 4: close the coroutine
>>> m.close()
10
Monday, November 28, 2011
Example: print out strings that match a pattern
def match(pattern):
print('Looking for ' + pattern)
execution starts
try:
while True:
resumes here
s = (yield)
s = “the Jabberwock ...”
if pattern in s:
print(s)
match found
except GeneratorExit:
catch exception
print("=== Done ===")
Step 1: Initialize
>>> m = match(“Jabberwock”)
does nothing
creates a new object
Step 2: Start with __next__()
>>> m.__next__()
‘Looking for Jabberwock’
Step 3: Send data
>>> m.send(“the Jabberwock with eyes of flame”)
‘the Jabberwock with eyes of flame’
Step 4: close the coroutine
>>> m.close()
‘=== Done ===’
Monday, November 28, 2011
10
Pipelines: the power of coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
11
Monday, November 28, 2011
Pipelines: the power of coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
We can chain coroutines together to achieve complex behaviors
11
Monday, November 28, 2011
Pipelines: the power of coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
We can chain coroutines together to achieve complex behaviors
Create a pipeline
11
Monday, November 28, 2011
Pipelines: the power of coroutines
coroutine
coroutine
coroutine
coroutine
coroutine
We can chain coroutines together to achieve complex behaviors
Create a pipeline
Coroutines send data to others downstream
11
Monday, November 28, 2011
A simple pipeline
read
words
match
words
12
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
def read(text, next_coroutine):
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
def read(text, next_coroutine):
for word in text.split():
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
match
words
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
13
Monday, November 28, 2011
A simple pipeline: reading words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
words
match
words
to know where to send()
13
Monday, November 28, 2011
A simple pipeline: reading words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
words
match
words
to know where to send()
read
for word in text.split():
next_coroutine.send(word)
13
Monday, November 28, 2011
A simple pipeline: reading words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
words
match
words
to know where to send()
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
match
words
to know where to send()
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
value = (yield)
next_coroutine
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
match
words
to know where to send()
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
loop
value = (yield)
next_coroutine
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
match
words
to know where to send()
read
for word in text.split():
next_coroutine.send(word)
(yield) -- wait for next send
send -- activate (yield)
loop
value = (yield)
next_coroutine
13
Monday, November 28, 2011
A simple pipeline: reading words
read
words
def read(text, next_coroutine):needs
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
match
words
to know where to send()
read
for loop
for word in text.split():
next_coroutine.send(word)
(yield) -- wait for next send
send -- activate (yield)
loop
value = (yield)
next_coroutine
13
Monday, November 28, 2011
A simple pipeline
read
words
match
words
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for word in text.split():
next_coroutine.send(word)
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
while True:
line = (yield)
if pattern in line:
print(line)
match
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
while True:
line = (yield)
if pattern in line:
print(line)
while loop
match
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
(yield) -- wait for next send
while True:
line = (yield)
if pattern in line:
print(line)
while loop
match
14
Monday, November 28, 2011
A simple pipeline
read
words
match
words
read
for loop
for word in text.split():
next_coroutine.send(word)
send -- activate (yield)
(yield) -- wait for next send
while True:
line = (yield)
if pattern in line:
print(line)
while loop
match
14
Monday, November 28, 2011
A simple pipeline
15
Monday, November 28, 2011
A simple pipeline
>>> matcher = match('ending')
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
>>> matcher = match('ending')
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
>>> matcher = match('ending')
>>> matcher.__next__()
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
15
Monday, November 28, 2011
A simple pipeline
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
Commending
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
Commending
matcher
‘ending’
paused
Commending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
Commending
matcher
‘ending’
Commending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
Commending
matcher
‘ending’
Commending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
spending
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
spending
matcher
‘ending’
paused
spending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
spending
matcher
‘ending’
spending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
spending
matcher
‘ending’
spending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
is
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
is
matcher
‘ending’
paused
is
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
is
matcher
‘ending’
is
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
offending
matcher
‘ending’
paused
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
offending
matcher
‘ending’
paused
offending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
offending
matcher
‘ending’
offending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
offending
matcher
‘ending’
offending
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
read
text = ‘Comm
matcher
‘ending’
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
last word!
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
text = ‘Comm
matcher
‘ending’
line = (yield)
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
last word!
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
text = ‘Comm
matcher
‘ending’
GeneratorExit
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
last word!
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
text = ‘Comm
matcher
‘ending’
GeneratorExit
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
last word!
‘=== Done ===’
15
Monday, November 28, 2011
A simple pipeline
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
text = ‘Comm
matcher
‘ending’
closed
GeneratorExit
>>> matcher = match('ending')
>>> matcher.__next__()
‘Looking for ending’
>>> text = 'Commending spending is offending to people pending lending!'
>>> read(text, matcher)
‘Commending’
‘spending’
‘offending’
‘pending’
‘lending!’
last word!
‘=== Done ===’
15
Monday, November 28, 2011
Produce, Filter, Consume
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
producer
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
producer
The producer
only sends data
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
send
filter
producer
(yield)
The producer
only sends data
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
send
filter
producer
(yield)
The producer
only sends data
The filter
consumes with (yield)
and sends results
downstream
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
send
filter
producer
...
(yield)
The producer
only sends data
The filter
consumes with (yield)
and sends results
downstream
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
filter
producer
(yield)
The producer
only sends data
send
send
...
filter
(yield)
The filter
consumes with (yield)
and sends results
downstream
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
filter
producer
(yield)
The producer
only sends data
send
send
...
filter
(yield)
The filter
consumes with (yield)
and sends results
downstream
There can be many
layers of filters
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
filter
producer
(yield)
The producer
only sends data
send
send
...
filter
(yield)
consumer
(yield)
The filter
consumes with (yield)
and sends results
downstream
There can be many
layers of filters
16
Monday, November 28, 2011
Produce, Filter, Consume
Coroutines can have different roles in a pipeline
Based on how they use send() and yield
send
filter
producer
(yield)
The producer
only sends data
send
send
...
filter
(yield)
The filter
consumes with (yield)
and sends results
downstream
consumer
(yield)
The consumer
only consumes data
There can be many
layers of filters
16
Monday, November 28, 2011
Example: simple pipeline
17
Monday, November 28, 2011
Example: simple pipeline
read
text = ‘Comm
17
Monday, November 28, 2011
Example: simple pipeline
Producer
read
text = ‘Comm
17
Monday, November 28, 2011
Example: simple pipeline
Producer
read
text = ‘Comm
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
17
Monday, November 28, 2011
Example: simple pipeline
Producer
read
text = ‘Comm
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
17
Monday, November 28, 2011
Example: simple pipeline
Producer
read
text = ‘Comm
matcher
‘ending’
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
17
Monday, November 28, 2011
Example: simple pipeline
Producer
Consumer
read
text = ‘Comm
matcher
‘ending’
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
17
Monday, November 28, 2011
Example: simple pipeline
Producer
Consumer
read
text = ‘Comm
matcher
‘ending’
def match(pattern):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
print(s)
except GeneratorExit:
print("=== Done ===")
17
Monday, November 28, 2011
Breaking down match
read words
match words
18
Monday, November 28, 2011
Breaking down match
Producer
read words
match words
18
Monday, November 28, 2011
Breaking down match
Producer
read words
Consumer
match words
18
Monday, November 28, 2011
Breaking down match
Producer
read words
Consumer
match words
18
Monday, November 28, 2011
Breaking down match
Producer
read words
Consumer
match words
find matches
print
18
Monday, November 28, 2011
Breaking down match
Producer
Consumer
read words
match words
find matches
print
filter
18
Monday, November 28, 2011
Breaking down match
Producer
Consumer
read words
match words
find matches
filter
print
consumer
18
Monday, November 28, 2011
Breaking down match
19
Monday, November 28, 2011
Breaking down match
find matches
print
19
Monday, November 28, 2011
Breaking down match
find matches
print
filter
19
Monday, November 28, 2011
Breaking down match
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
find matches
print
def match_filter(pattern, next_coroutine):
filter
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
find matches
filter
print
consumer
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
>>> printer = print_consumer()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
>>> printer = print_consumer()
>>> printer.__next__()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
‘Looking for pend’
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
‘Looking for pend’
>>> text = 'Commending spending is offending'
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
‘Looking for pend’
>>> text = 'Commending spending is offending'
>>> read(text, matcher)
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
‘Looking for pend’
>>> text = 'Commending spending is offending'
>>> read(text, matcher)
‘spending’
19
Monday, November 28, 2011
Breaking down match
find matches
def match_filter(pattern, next_coroutine):
print('Looking for ' + pattern)
try:
while True:
s = (yield)
if pattern in s:
next_coroutine.send(s)
except GeneratorExit:
next_coroutine.close()
filter
print
consumer
def print_consumer():
print('Preparing to print')
try:
while True:
line = (yield)
print(line)
except GeneratorExit:
print("=== Done ===")
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> matcher = match_filter('pend', printer)
>>> matcher.__next__()
‘Looking for pend’
>>> text = 'Commending spending is offending'
>>> read(text, matcher)
‘spending’
‘=== Done ===’
19
Monday, November 28, 2011
Multitasking
coroutine
coroutine
coroutine
coroutine
coroutine
20
Monday, November 28, 2011
Multitasking
coroutine
coroutine
coroutine
coroutine
coroutine
We do not need to be restricted to just one next step
20
Monday, November 28, 2011
Read-to-many
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
read
coroutine
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
21
Monday, November 28, 2011
Read-to-many
def read(text, next_coroutine):
for word in text.split():
next_coroutine.send(word)
next_coroutine.close()
def read_to_many(text, coroutines):
for word in text.split():
for coroutine in coroutines:
coroutine.send(word)
for coroutine in coroutines:
coroutine.close()
read
coroutine
coroutine
read_to_many
coroutine
21
Monday, November 28, 2011
Matching multiple patterns
read_to_many
Any questions?
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> m = match_filter('mend', printer)
>>> m.__next__()
‘Looking for mend’
>>> p = match_filter("pe", printer)
>>> p.__next__()
‘Looking for pe’
>>> read_to_many(text, [m, p])
‘Commending’
‘spending’
‘people’
‘pending’
‘=== Done ===’
22
Monday, November 28, 2011
Matching multiple patterns
match
‘mend’
read_to_many
Any questions?
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> m = match_filter('mend', printer)
>>> m.__next__()
‘Looking for mend’
>>> p = match_filter("pe", printer)
>>> p.__next__()
‘Looking for pe’
>>> read_to_many(text, [m, p])
‘Commending’
‘spending’
‘people’
‘pending’
‘=== Done ===’
22
Monday, November 28, 2011
Matching multiple patterns
match
‘mend’
read_to_many
match
‘pe’
Any questions?
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> m = match_filter('mend', printer)
>>> m.__next__()
‘Looking for mend’
>>> p = match_filter("pe", printer)
>>> p.__next__()
‘Looking for pe’
>>> read_to_many(text, [m, p])
‘Commending’
‘spending’
‘people’
‘pending’
‘=== Done ===’
22
Monday, November 28, 2011
Matching multiple patterns
match
‘mend’
print
read_to_many
match
‘pe’
Any questions?
>>> printer = print_consumer()
>>> printer.__next__()
‘Preparing to print’
>>> m = match_filter('mend', printer)
>>> m.__next__()
‘Looking for mend’
>>> p = match_filter("pe", printer)
>>> p.__next__()
‘Looking for pe’
>>> read_to_many(text, [m, p])
‘Commending’
‘spending’
‘people’
‘pending’
‘=== Done ===’
22
Monday, November 28, 2011
NEXT TIME
MAP REDUCE
http://www.infobarrel.com/Top_10_Tips_For_Snowboard_Beginners
Monday, November 28, 2011