Topics in Computer Science: COMS E6998-07-2015-03 Micro-service Application and API Development Lecture 1: Overview, Concepts, REST Dr. Donald F. Ferguson [email protected] (Admin: [email protected]) © Donald F. Ferguson, 2015. All rights reserved. Contents 2 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Contents • Overview – – – • 3 Basic concepts Traditional realization Transactions, REST – – • • 35 min. Evolution to micro-services. Micro-services concepts. APIs. Datamodel – – – • A little bit about me. Initial lecture schedule. Coursework and grading. Concepts – – – • 15 min. Concepts and overview. Some best practices. 1st Project and related concepts Q&A, discussion Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST 10 min. © Donald F. Ferguson, 2015. All rights reserved. Microservice Concepts (Cont) 4 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Monolithic to Micro Cart Functions • • Java SQLite Recommendation Functions • Node.js • Redis Catalog Functions • • PDP MongoDB XXX XXX • • MMM NNN Content Functions • • 5 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Ruby Amazon S3 © Donald F. Ferguson, 2015. All rights reserved. SOA vs Microservices 6 http://www.pwc.com/us/en/technology-forecast/2014/cloudcomputing/features/microservices.jhtml Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Inject application implementation into reusable SW containers Reusable infrastructure containers Reusable SW containers but with core technology and frameworks 7 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. 8 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. API Economy Sometime you just reuse an API instead of building a micro-service. 9 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. The Course – We Will Cover • Patterns, technology and best practices for – Implementing base microservices. – Assembling microservices into “solutions.” – Achieving scalability, availability, agility, etc. • Web callable infrastructure services that accelerate implementing microservices, e.g. – Databases – Messaging – …… • Web callable application/business services that accelerate implementing a microservise’s business/domain functions. 10 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Microservice Patterns New Microservice Patterns http://microservices.io/patterns/ 11 Classic Patterns, New Realization “Enterprise Integration Patterns,” Fowler et al. Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Backup 12 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. What can we Learn from Data 13 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Monolithic to Micro Cart Functions • • Java SQLite Recommendation Functions • Node.js • Redis Catalog Functions • • PDP MongoDB XXX XXX • • MMM NNN Content Functions • • 14 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Ruby Amazon S3 © Donald F. Ferguson, 2015. All rights reserved. Database Model are Complex, even examples and samples, e.g. MySql Sakila Sample Database 15 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Some Concepts • Foreign key relationships – Represent 1-1, 1-N relationships – Have “behavior,” e.g. – – On Delete would prevent deleting a country if there is a city whose country_id is the country’s id. On Cascade would automatically update all city.country_id when country.id changes • Defining indices is important to avoid – Scanning the entire city table to – To find cities in a given country • I would not put strings in a table for most words and string – Putting “Spain” for a country name – Prevents localization and national language enablement – Use symbols into localization resource bundles 16 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Views and Stored Procedures 17 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. What can we Learn? The implementation of the SOA contract/REST interface – Is a set of verbs on URLs – That manipulate a logical data model. • Every logical data model has a common set of concepts that materialize through REST – ID URI/URL – Collections supporting – – – Primary key …/Customers/21 Non-unique, secondary keys …/Customers/Zipcode/12345 Ad hoc query (SELECT WHERE (… …)) …/Customers?q=“id<=50&lastname=Ferguson” – Projection – – SELECT iq, lastname FROM Customers …/Customers?”Fields=iq,lastname” UPDATE iq, shoessize WHERE … PUT {{iq, “50},{…}} …/Customers – Foreign keys/join tables Hyperlinks – Iterators – – – – – – 18 SELECT * FROM Customers CREATE Cursor … GET …/Customers?Offset=40&Pagesize=20 Thread/callback/promise Asynchronous REST responses Metadata/reflection: SQL DESCRIBE TABLE Web UI for driving the REST API Stored procedures PUT…/Commands/… Events/Notifcations Feeds Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Some Concerns 19 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. “Decomposing applications for deployability and scalability,” Chris Richardson, http://plainoldobjects.com/ 20 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Simple to manage • Atomicity • Consistency • Isolation • Durability 21 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. ACID Transactions 22 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. ACID Transactions • ACID transactions – Atomic: A set of writes all occur or none occur. – Durable: The data “does not disappear,” e.g. write to disk. – Consistent: My applications move the database between consistent states, e.g. the transfer function works correctly. • Isolation – Determines what happens when two or more threads are manipulating the data at the same time. – And is defined relative to where cursors are and what they have touched. – Because the cursor movement determines what you are reading or have read. 23 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Monolithic to Micro Cart Functions • • Java SQLite Recommendation Functions • Node.js • Redis Catalog Functions • • PDP MongoDB XXX XXX • • MMM NNN ACID is pretty straightforward • When there is a logical, single DB. • With integrated code moving from consistent state to consistent state. But, this gets really hard when • Data is federated across multiple, independent microservices and DBs. • With different approaches to transactions, replication, scalability, … … 24 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Content Functions • • Ruby Amazon S3 © Donald F. Ferguson, 2015. All rights reserved. CAP Theorem The CAP theorem, also known as Brewer's theorem, states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: • Consistency (all nodes see the same data at the same time) • Availability (a guarantee that every request receives a response about whether it succeeded or failed) • Partition tolerance (the system continues to operate despite arbitrary partitioning due to network failures) 25 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. REST Introduction 26 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Representational State Transfer (REST) • People confuse – Various forms of RPC/messaging over HTTP – With REST • REST has six core tenets – – – – – – 27 Client/server Stateless Caching Uniform Interface Layered System Code on Demand Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. REST Tenets • Client/Server (Obvious) • Stateless is a bit confusing – The server/service maintains resource state, e.g. Customer and Agent info. – The conversation is stateless. The client provides all conversation state needed for an API invocation. For example, – customerCursor.next(10) requires the server to remember the client’s position in the iteration through the set. – A stateless call is customerCollection.next(“Bob”, 10). Basically, the client passes the cursor position to the server. • Caching – The web has significant caching (in browser, CDNs, …) – The resource provider must – Consider caching policies in application design. – Explicitly set control fields to tell clients and intermediaries what to cache/when. 28 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. REST Tenets • Uniform Interface – Identify/locate resources using URIs/URLs – A fixed set of “methods” on resources – – myResource.deposit(21.13) is not allowed The calls are – – – – Get Post Put Delete – Self-defining MIME types (Text, JSON, XML, …) – Default web application for using the API – URL/URI for relationship/association • Layered System: Client cannot tell if connected to the server or an intermediary performing value added functions, e.g. – Load balancing – Security – Idempotency • Code on Demand (optional): Resource Get can deliver helper code, e.g. – JavaScript – Applets 29 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. SSOL Page 30 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Anatomy of a URL • SSOL for the Classlist https://ssol.columbia.edu/cgibin/ssol/DhbtiwfsFOMOeFQaDwqxAh/?p%.5Fr%.5Fid=k0F2vZ4ccAhzbcAg0Ql K4h&p%.5Ft%.5Fid=1&tran%.5B1%.5D%.5Fentry=student&tran%.5B1%.5D%. 5Fterm%.5Fid=20143&tran%.5B1%.5D%.5Fcid=COMSE6998&tran%.5B1%.5 D%.5Fsecid=005&tran%.5B1%.5D%.5Fsch=&tran%.5B1%.5D%.5Fdpt=&tran %.5B1%.5D%.5Fback=&tran%.5B1%.5D%.5Ftran%.5Fname=scrs • This is – Not REST – This is some form of Hogwarts spell – This is even bad for a web page 31 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Anatomy of a URL • HTTP://www.somedomain.edu/... The “server” container • …/ssol/… The module/component • …/listManager …/Class/COMSE6998-01 The Application Object or Entity Class (“Extent”) and ID • .../WaitingList/… Contained Resource • GET, POST, … on URL for CRUD • Some details – …/WaitlingList/dff9/IQ Path navigation into resources – …/WaitlingList?op=“Approve”?CUID=“dff9” Method 32 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Asynchronous Operation 33 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Implementation Observations • Define a collection /QueuedResponses – A client can call …/QueuedResponses/21 to get a specific response. – You already know how to do this for …/Customer – The data format in the table is {id, status, JSONString} • A simple implementation would be writing a façade – – – – – Accept request Create new table entry with status = “in progress” Return 202 and URL Call the actual implementation Update the database table entry with the JSON result • Most application platforms have middleware approaches to support registering callbacks, threads, etc. The implementation would typically – Invoke some long running action, e.g. DB query, workflow process and register a callback – The callback implementation updates the entry in the response table. 34 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Ad Hoc Query • Every collection should support ?q=“… …” – …/Customers?q=“lastName=21&IQ<21” – q is a string encoding a set of triplets with elements – Resource field, e.g. “lastName” – Comparison operation, e.g. “=“, “>”, … – Comparison value. • Your code needs to – Parse and validate the query string. – Rewrite the string in the query language of the underlying database, e.g. Where clause in SQL – Execute the query – Refine the result set if the underlying database does not support query capabilities that you are surfacing through your API. 35 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Pagination {“data”: [{“user_id”:”42”, “name”:”Bob”, “links”:[{“rel”:”self”, “href”:”http://api.example.com/users/42”}]}, {“user_id”:”22”, “name”:”Frank”, “links”: [{“rel”:”self”, “href”:”http://api.example.com/users/22”}]}, {“user_id”:”125”, “name”: “Sally”, “links”:[{“rel”:”self”, “href”:”http://api.example.com/users/125”}]}], “links”: [{“rel”:“first”, “href”:”http://api.example.com/users?offset=0&limit=3”}, {“rel”:“last”, “href”:”http://api.example.com/users?offset=55&limit=3”}, {“rel”:“previous”, “href”:”http://api.example.com/users?offset=3&limit=3”}, {“rel”:”next”, “href”:”http://api.example.com/users?offset=9&limit=3”}]} 36 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Implementation Considerations • Query rewrite – …/Customers?q=“lastname=Ferguson&id<5”&limit=10&offset=5 – Neatly translates into an SQL statement – Select * from customers where … limit=5 offset=5 • Other databases have similar concepts. You may have to – Rewrite a push the query down – Build a result cache in another store that supports limit/offset – Paginate through the cache • You should also consider adding – “field=lastname,IQ,color” – To enable selecting a subset of fields 37 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Some Complex Topics 38 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Example Considerations • Service Endpoint – – • Requests and Authentication – – – – • Through HTTP Link Header Syntax and Example of Pagination Link Header Consistency Across Page Requests Versioning – – 39 Atomic Types Complex Type - Object or Structure Resource Relationships Resource References Pagination – – – • Resource Requests Resource Representation Resource Methods Synchronous Operations Asynchronous Operations Success Response Codes Failure Response Codes • Saying “REST is not enough • You have to define a set of patterns/ conventions of URLs, headers, … Resource Data Types – – – – • Request Headers Request Timestamps Request Authentication Response Headers Resources – – – – – – – • Endpoint Endpoint Encryption Version Header Version URI Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Endpoint • <URL>/a/b/c?x=7&y=21 is a pretty straightforward concept, but … – How do I get info about customer “Ferguson?” – …/Customer/Ferguson/Donald – …/Customer?lastName=“Ferguson”&firstName=“Donald” – ??? – Do I really want to – Find info about Don using …/Customer/Ferguson/Donald – Find info about agent using …/Agents?id=“21” – How does it work if I can find customer by name or phone number? – How do I set a relationship between customer and agent? – PUT …/Relationship/AgentFor?agent=“21”&”Customer=“Ferguson” – Or two PUTS, one on Customer and one on Agent? • In the same way you have to define a framework for your application, you have to define a shape/pattern in your REST API model. 40 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Headers – Some Choices I Made HTTP Request Header 41 Value Mandatory auth-timestamp: The current POSIX time. Yes auth-key: The user or client’s unique API KEY. Yes auth-signature: The HMAC-SHA256 digest for Yes the request. api-version: (Optional) API version string No Accept: (Optional) application/xml or application/json No Nonce: One time UUID to enable idempotency/duplicate detection Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Response Codes Operation READ HTTP Request GET HTTP Response Codes Supported 202 means 200 - OK with message body 204 - OK no message body 206 - OK with partial message body CREATE POST 201 - Resource created (Operation Complete) 202 - Resource accepted (Operation Pending) UPDATE PUT 202 - Accepted (Operation Pending) 204 - Success (Operation Complete) DELETE DELETE 202 - Accepted (Operation Pending) 204 - Success (Operation Complete) • Your request went asynch. • The HTTP header Link is where to poll for rsp. Examples of Link Headers in HTTP response: Link: <http://api/jobs/j1>;rel=monitor;title="update profile" Link: <http://api/reports/r1>;rel=summary;title=”access report” 42 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Failure Response Code Error 43 Response Code Invalid Parameter 400 - Invalid parameter Authentication 401 - Authentication failure Permission Denied 403 - Permission denied Not Found 404 - Resource not found Invalid Request Method 405 - Invalid request method Internal Server Error 500 - Internal Server Error Service Unavailable 503 - Service Unavailable Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Design Pattern Choices Resource Requests Collection https://ENDPOINT/NAMESPACE/RESOURCE[?QUERY_ PARAMETERS] Resource https://ENDPOINT/NAMESPACE/RESOURCE/RESOURCE_ ID[?QUERY_ PARAMETERS] Collection Operation Support for • Map • Array • Collection Get all items in the collection GET /collection Get an particular item in the collection Should also return the URI of the collection itself. GET /collection/itemId Get items match certain criteria GET /collection?property1=’value’ Add a new item to the collection POST /collection Get items starting at 100 with page size=25 44 HTTP Request Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST contents of new item … GET /collection?start=100&pageSize=25 © Donald F. Ferguson, 2015. All rights reserved. Relationships Relationship as a resource {“membership”: { “URI” “created” “owner” “expire” “group” “server” } } : : : : : : “http://dell.com/memberships/m12356”, “2013-08-01T12:00:00.0Z”, “user123456”, “never”, { “ref” : “http://dell.com/groups/g123456” }, { “ref” : “http://dell.com/servers/s123456”} Relationship as a field in resource "link": { "href": "http://dell.com/api/resource1", "rel": "self", “title” : “server-s123456” } 45 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Navigating Through Result Set • GET on collections, maps, … needs pagination and cursors – Limit: What is the maximum number of elements you want? – QueryID: A tag for the query that produced the original result set – Offset references a specific element in a “page.” • There is a standard for linking resources in logical sets, e.g. – Link: <http://example.com/TheBook/chapter2>; rel="previous"; title="previous chapter“> – Indicates that "chapter2" is previous to this resource in a logical navigation path. • Your API/framework can use this for result sets – Example 1 – – – GET /api/customers?status=“Gold” returns some number of “Gold” customers and Link Header for “next page” is Link: </api/customers?status=“Gold”&offset=50&limit=50>; rel="next last“ Which is the URL for the “cursor.next set,” which has 50 elements and is also “last” – Example 2 – – – 46 Get </api/customers?status=“Gold”&offset=50&limit=50> returns the “next” from example 1 With Link Link: </api/asm/servers?status=ready&limit=50>; rel="prev first" Allowing you to go backwards to the previous “page.” Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Components Assembly 47 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. SOA Component Architecture Microservices 48 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. SOA Component Architecture Microservices Five elements still matter 1. Interface and binding. 49 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. 1st Assignment 50 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Q&A Discussion 51 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Backup 52 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. 53 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. X-Axis and Y-Axis 54 Scale-Out through • • Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST Shared database Cloned, stateless code © Donald F. Ferguson, 2015. All rights reserved. 55 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. • 1st Projects – Simple content management – Mail verification for registration; CAPTCH 56 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Scalability – Performance, Development, … 57 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved. Scaling https://devcentral.f5.com/articles/the-art-of-scale-microservices-the-scale-cube-and-load-balancing 58 Micro-service Application and API Development (E6998-07-2015-03) – Lecture 1: Overview, Concepts, REST © Donald F. Ferguson, 2015. All rights reserved.
© Copyright 2026 Paperzz