REST Web Services oAuth2 Tutorial - Felix John COLIBRI.

• abstract : Delphi DropBox Rest Service Client using the OAuth2 protocol. Implemented with the tRestClient Delphi components or Indy tIdHttp component. Get the DropBox token, list the files, download and upload files
  
• key words : Delphi Rest Client, DropBox; oAuth2 protocol, tRestClient tRestRequest tRestResponse, Indy tItHttp, OpenSSL
  
• hardware used : intel i3-8100, 3.6 gHz, 8GB memory, 128 G and 1 T hard disc
  
• software used : Windows 10 64 bit, Delphi Xe8
  
• scope : Delphi 1 to 7, 2006 to 2010, Xe, Seattle, Tokyo, Berlin, 10.3
  
• level : Delphi developer
  
• plan :
  
  • DropBox App
    
  • The OAuth2 protocol
    
  • Creating the DropBox App
    
  • The DropBox API
    
  • The Delphi DropBox Rest Client
    
  • The Indy DropBox Rest Client
    
  • Comments
    
  • Download the Sources
    
  • References and links
    

1 - DropBox App

This is where Oauth2 comes into play.

This article will present

• the Oauth2 teminology
  
• the Oauth2 flow
  
• how to create a DropBox App
  
• the Delphi DropBox application allowing to download / upload files to this DropBox account
  

2 - The OAuth2 protocol

2.1 - The 3 parties

• DropBox which allows anybody to create an account and save data on the Cloud and retrieve this data from anywhere. This is the SERVICE
  
• the person which creates this DropBox account. He is the Oauth2 (account) OWNER, or sometimes called the "user"
  
• some other person, the CLIENT, might want to read the OWNER data
  

2.2 - OWNER creates an account and puts some data

and then puts some data in his account

2.3 - OWNER creates a SERVICE APP

Using some SERVICE utility, he requests a "SERVICE App". The SERVICE generates a Client_id, a Secret_id and asks for a redirection URL.

So the OWNER must have an HTTP server (usually HTTPS) where SERVICE will be able to redirect the authorization requests. Suppose the url of this server is https//redir.html.

The OWNER then creates the "SERVICE App" After this operation, the OWNER account looks like this

CLIENT requests token

Some CLIENT wants to read the OWNER data. To do so he will use a REST service asking SERVICE for this data. He will be allowed to do so if he has a token.

To get this token

• the CLIENT asks the OWNER to send him (by phone, mail, email etc) the client_id and the redirect.html

• the client sends a REST request to the SERVICE to get the token

• on the SERVICE, this REST request is handled by an authentication server. This server uses the client_id to locate the "OWNER App". It checks that the redir.html matches the redir.html the OWNER provided. If this is the case, the SERVICE sends an html Form to the OWNER https://redir.html, asking him whether he accepts this request

• the OWNER sees an HTML page and click "OK". This is sent back to the SERVER

• the SERVICE then generates a token and sends it back to the CLIENT

• with this token, the CLIENT can now performed any action the SERVICE allows. In our case, then CLIENT will request the data

• happy to oblige, the SERVICE sends back the data

• as the diagram shows, the OWNER credential are never exchanged during this protocol. The purpose of OAUTH2 is to replace the OWNER userpassword with the token
  
• the token is generated by the SERVICE when required. Its lifetime depends on the SERVICE. For Dropbox, it lasts about 2 hours. After that the CLIENT has to travel this route again. In fact, there are refresh tokens, which allow the CLIENT to use the token for a longer time span.
  On the DropBox App page, the OWNER can create a new token by clicking the "Create Token" button. But the OWNER is not supposed to hand this token to the CLIENT if he follows the Oauth2 protocol. However we can use it for debugging the CLIENT application, to avoid the tedious and long acceptation dialog (steps 6 to 9)

• steps 6 to 9 are used to obtain this token. Steps 10 and 11 are then repeated by the CLIENT as many time as he likes, and allow him to send the whole gammut of services the SERVICE offers. For DropBox, we can require the list of the files, download files, upload files etc
  
• when the OWNER creates the "SERVICE App", he has no idea of the application the CLIENT is going to use. It can be a Delphi application, the Delphi RestDebugger, a Java application. In fact any application able to send client Rest requests and handle the Oauth2 protocol.
  
• for me, the name "client_id" is a misnomer: this ID is created by the SERVICE when the OWNER creates the "SERVICE App". I would have called this "owner_id". The same goes for "SERVICE App" (like a DropbBox App). It is not an application at all, but a record with 4 informations (client_id, secret_id, redirection_url and token). The CLIENT will create or use some application
  

    "OAUTH2 is a protocol that allows a user to grant limited access to their resources on one site, to another site, without having to expose their credentials".

And

• the OWNER only sends the client_id and redirection_url to the CLIENT. Not his DropBox User / Password
  
• with those client_id / redirection_url, the CLIENT can ask a token, and then use this token to read from and write to the DropBox
  

3 - Creating the DropBox App

Next, to be able to upload and download files, you must create a "DropBox App"

	go to the Dropbox developer page https://www.dropbox.com/developers and click "Create apps"
	the app creation page is displayed
	select "Dropbox api"
	the access selection buttons are displayed
	select "Full Dropbox" name your app, like "delphi_boxer_app", check "I agree" and click "Create app"
	the app settings page is displayed
	save your key     mshall8oznmrrcy click on "secret" to display and save your secret key     p6j8ef54yyjy8du type the URL of your redirection     http://localhost/ and click "Add" click "generate" to generate your access token, and save it     X5SWPFqX3CAAAAAAAAAAL6TPIHZYgVSCIw4OPOgiIvibZTpMqNoobY5cXMSBxU0l

Note

• to delete a redirection url, select it's line and click the "x" at the right of the line
  

3.1 - Error

we could not retrieve the redirection page

3.2 - Display your apps

	in your browser, type the URL https://www.dropbox.com/developers/apps
	all your apps are displayed

4 - The DropBox API

4.1 - How to find the Api documentation

So typing "DropBox" and clicking "Search Apis" will display a DropBox link:

Clicking on the DropBox link we get

This link brings us to DropBox V1 documentation:

Now since I know that DropBox v1 should be replaced by DropBox v2, I found the correct DropBox V2 documentation:

On the right of the page are the links to the different commands. The documentation for each command is quite complete, For example, to download a file:

to get our token, we select

5 - The Delphi DropBox Rest Client

5.1 - Delphi RestDemo

I simply extracted the DropBox part and placed it in our tRestClient demo.

5.2 - The tRestClient DropBox project

Clicking on "get_token_" opens the redirection browser, and the steps are detailed below with the Indy version of the Rest Client.

We then tried to use our token, by clicking "fetch_metadata_". The code is beautifully simple:

However the result was a "400 bad request" with RestResponse.Content telling "Unknown API function: "metadata/dropbox/".

I tried to download files or upload files, the first resulted in "Unexpected URL params: "access_token"", the second in an access violation. Spending some time exploring the Delphi Rest architecture would have solved those problems. But I decided instead to use Indy which allows more direct control over the HTTP code.

6 - The Indy DropBox Rest Client

6.1 - Objective

Of course, since, as I understand, the Delphi Rest framework (and the DataSnap framework) rest (pun intended) on the Indy Server. So whatever logging you can accomplish using the basic tIdHttpServer should be possible with the Delphi Rest Framework. Finding out how to do so proves to be somewhat difficult to accomplish however.

6.2 - Getting the DropBox access token

The application looks like this

Here ie the procedure asking the access token:

	we click the "get_access_token_direct_" button The code is the following Procedure TForm1.get_access_token_direct_Click(Sender: TObject);   Const k_get_access_code_url=               'https://www.dropbox.com/1/oauth2/authorize'             + '?client_id=%s'             + '&response_type=%s'             + '&redirect_uri=%s';   Var l_dropbox_authorize_url: string;       l_get_result: String;   Begin     dropbox_access_token_edit_.Text:= '';     display('> get_token_direct_Click');     l_dropbox_authorize_url:= Format(k_get_access_code_url,         [tIdURI.PathEncode(k_client_id), tIdURI.PathEncode('token'),          k_redirection_uri]);     display('url '+ l_dropbox_authorize_url);     m_last_title:= '';     m_last_url:= '';     If Not execute_.Checked       Then Exit;     display_pagecontrol.ActivePage:= browser_tabheet_;     // -- this starts the Browser asking the access code     WebBrowser1.Navigate(l_dropbox_authorize_url);     display('< get_token_direct_Click');   End; // get_token_direct_Click The following URL will be built (new line added)   https://www.dropbox.com/1/oauth2/authorize?   client_id=lyto17f3j3rr120   &response_type=token   &redirect_uri=http://localhost:3000 The WebBrowser will be started with this URL
	DropBox redirects to our redirection URL and asks for the user / password. We enter our credentials
	then we click "Sign in"
	DropBox warns us
	we accept the risk by clicking "Continue"
	DropBox asks the OWNER whether he allows the "delphi_boxer_app" to receive the access token
	we click "Allow"
	the application displays the access token in the dropbox_"access_token_edit_ and switches back to the log tabsheet (aka "closes the redirection browser")

We can follow the dialog since we create

• the OnTitleChange tWebBrowser event to see when we receive a page from DropBox

  Procedure TForm1.WebBrowser1TitleChange(ASender: TObject;     Const Text: WideString);   Begin     If (Text <> m_last_title)       Then Begin         display_line;         display('new_title '+ Text);         m_last_title := Text;       End;   End; // WebBrowser1TitleChange

• then NavigateComplete2 which will contains the access token

  Procedure TForm1.WebBrowser1NavigateComplete2(ASender: TObject;     Const pDisp: IDispatch; Const URL: OleVariant);   Var l_access_token_position: integer;       l_access_token: string;       l_ansi_token: AnsiString;   Begin     m_last_url := VarToStrDef(URL, '');     l_access_token_position := Pos('access_token=', m_last_url);     If l_access_token_position > 0       Then Begin           l_access_token := Copy(m_last_url, l_access_token_position + 13,               Length(m_last_url));           If (Pos('&', l_access_token) > 0) Then           l_access_token := Copy(l_access_token, 1, Pos('&', l_access_token) - 1);       End;   End; // WebBrowser1NavigateComplete2

  
  

So we simply look for the "access_token" keyword to extract this token, and save it for later reuse

And here is the log of the events

6.3 - Get The Code then the Access Token

As I understand, we should rather first ask DropBox for a "code", and then asks DropBox to convert this "code" into the "access token".

Here is how:

• the request procedure for the code is

  Procedure TForm1.get_code_Click(Sender: TObject);   Const k_get_url_2= 'https://www.dropbox.com/1/oauth2/authorize?'            + 'client_id=%s'            + '&response_type=code'            + '&redirect_uri=%s';   Var l_get_code_url: String;   Begin     display('> get_code');     l_get_code_url:= Format(k_get_url_2,       [k_client_id, tIdUri.PathEncode(k_redirection_uri)]);     display(l_get_code_url);     m_last_title_2:= '';     m_last_url_2:= '';     ClipBoard.AsText:= 'drop.owner@free.fr';     display_pagecontrol.ActivePage:= code_brownser_tabsheet;     WebBrowser2.Navigate(l_get_code_url);     display('< get_code');   End; // get_code_Click

• clicking on "get_code_" brings us to the same 3 DropBox pages, with at the end the OnNavigateComplete2 URL:

  http://localhost:3000/?   code=X5SWPFqX3CAAAAAAAAAApsdRO1Hs606iQFcTAkFRNuI

• then we click "get_token_from_code_"

  Procedure TForm1.get_token_from_code_Click(Sender: TObject);   // --  requests the code   //      curl https://api.dropbox.com/oauth2/token \   //       -d code=<AUTHORIZATION_CODE> \   //       -d grant_type=authorization_code \   //       -d redirect_uri=<REDIRECT_URI> \   //       -u <APP_KEY>:<APP_SECRET>   Const k_get_access_token_from_code_url= 'https://api.dropboxapi.com/oauth2/token';   Var l_c_id_http: TIdHTTP;       l_json: String;       l_c_utf8_json_string_stream: tStringStream;       l_post_response: String;   Begin     display('> get_token_from_code_Click');     l_c_id_http := TIdHTTP.Create(Nil);     With l_c_id_http Do       Try         HandleRedirects := False;         OnRedirect:= handle_access_token_redirect;         ConnectTimeout := 1000; // FHTTPTimeout;         Request.BasicAuthentication := False;         IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(l_c_id_http);         Request.Accept := 'application/x-www-form-urlencoded';         Request.ContentType := 'application/json';         OnRedirect:= handle_access_from_token_redirect;         l_json:= '{ "code":"' + f_dropbox_code_token + '", '+ k_new_line                + ' "grant_type":"authorization_code", '+ k_new_line                + ' "redirect_uri":"'+ tIdUri.PathEncode(k_redirection_uri)+ '", '+ k_new_line                + ' "client_id":"'+ k_client_id +'", '+ k_new_line                + ' "client_secret":"'+ k_secret_id +'" }';         display_formatted_json(l_json);         l_c_utf8_json_string_stream := TStringStream.Create(l_json, TEncoding.UTF8, True);         Try           display('--- send post');           l_post_response := l_c_id_http.Post(k_get_access_token_from_code_url,                l_c_utf8_json_string_stream);           display('--- post_result');           display(l_post_response);         Finally           l_c_utf8_json_string_stream.Free;         End;       Finally         l_c_id_http.Free;       End;     display('< get_token_from_code_Click');   End; // get_token_from_code_Click

• the Json is the following

  {     "code":"3CAAAAAAAAAApsdRO1Hs606iQFcTAkFRNuI",     "grant_type":"authorization_code",     "redirect_uri":"http://localhost:3000",     "client_id":"lyto17f3j3rr120",     "client_secret":"6u9c630qw7tgy72" }

• at this stage we should have received the access token. Sadly we got the following exception

  

  The Indy log is the following

  Stat Connected. Sent 02/12/2019 08:33:55:     POST /oauth2/token HTTP/1.0     Content-Type: application/json     Content-Length: 201     Host: api.dropboxapi.com     Accept: application/x-www-form-urlencoded     Accept-Encoding: identity     User-Agent: Mozilla/3.0 (compatible; Indy Library) Sent 02/12/2019 08:33:55:     { "code":"3CAAAAAAAAAApsdRO1Hs606iQFcTAkFRNuI",     "grant_type":"authorization_code",     "redirect_uri":"http://localhost:3000",     "client_id":"lyto17f3j3rr120",     "client_secret":"6u9c630qw7tgy72" } Recv 02/12/2019 08:33:55:     HTTP/1.1 400 Bad Request     Server: nginx     Date: Mon, 02 Dec 2019 07:33:59 GMT     Content-Type: application/json     Connection: close     Content-Security-Policy: sandbox; frame-ancestors 'none'     X-Dropbox-Request-Id: 3c6874c6aa49211bdad2692dbaf9c895     X-Frame-Options: DENY     X-Content-Type-Options: nosniff     Content-Disposition: attachment; filename='error'     {"error_description": "No auth function available for given request",     "error": "invalid_request"} Stat Disconnected.

  I tried to Encode64 the client_id and the secret_id, but without success.

  Hopefully some reader will be kind enough to tell me where I goofed.
  

So now we can proceed with the standard DropBox file management routines: listing the files, downloading or uploading some file etc.

6.4 - Listing our DropBox files

The (truncated) Json response is

Of course we could also look at the Indy log to examine the full TCP / IP exchange.

6.5 - Downloading a file from DropBox

Here is the code:

This took me some time to understand. I first believed that the file would be saved by the tFileStream. Yet the result was always a 0 byte file. It turns out that the file is resurned as the tIdHttp.Post function. Therefore we saved this string to disk.

6.6 - Upload a file to DropBox

This is our procedure :

7 - Comments

7.1 - DropBox

• in the DropBox credential page, there si a suggestion "Sign in with Google". I recommend NOT TO DO SO. To avoid the typing of the user / password, I did it, and then Google warned me, on the PC and on my phone, that DropBox had full access to my google account. So I changed my Google password and no longer used this option.

• the "Url Endpoints" are not very consistent. Sometimes www.dropbox.com. But other times api.dropboxapi.com. The best is to look at the DropBox documentation

• to write our DropBox App, we used a localhost IP. In this case, and this case only, your redirection URL can be HTTP. In production applications, it must be HTTPS.

• we created the drop.owner@free.fr mail. Then I wanted to call the App "dropbox_xxx". However DropBox forbids it. The name MUST not contain "dropbox". This is why we called our App "delphi_boxer_app"

• beware that DropBox now uses a Version 2 api, so most (not all) URIs use the "/2/" segment instead of the old "/1/" segment (which you still may encounter in some of the examples downloaded from Google

• during my Google Oauth2 journeys, I came across some posts claiming that DropBox was not "totally Oauth2". Whatever that means ...
  

7.2 - Our Indy Implementation

• to avoid typing the credentials, in my trials, before calling the tWebBrowser.Navigate I used

  ClipBoard.AsText:= 'drop.owner@free.fr';

  When the Browser page was presented, I simply pasted the email in the page email edit. Then I clicked the "pass" button which copies my password to the clipboard and pasted that to the Page password edit.

  Of course hardoding the user / password in the code is not something one would recommend. In fact it runs against the basic principle of Oauth2, where the application does never see the user / pass and uses this convoluted redirection scheme to let the OWNER type his credentials.

• about the extraction of the access token or the code, we used the tWebBrowser.OnAfterExecute2 event. The Delphi RestDemo calls a Tfrm_OAuthWebForm which contains the tWebBrowser. This Form is located in

      sources\data\rest\REST.Authenticator.OAuth.WebForm.Win.pas

  The WebBrowser implements the onTitleChange, OnBeforeExecute2 and onAfterExecute2 events. Those events can call our application callbacks to extract whatever information we want from the different Rest Servers. In our case, DropBox uses the onAfterExecute2 event to send the access token or the code back, but I imagine some other Rest Servers have other techniques.

  In the callbacks, once the relevant information has been extracted (the access token, for instance), the callback uses the VAR pv_do_close_the_browser parameter which is used by tWebBrowser event to close the WebBrowser.

  It seems that closing the redirected page was a major issue in the Oauth2 implementation, because the user would have this Browser page still open after having answered to all the questions.

  In our Indy implementation, we simply change the Tabsheet page once we got the DropBox answer.

  And we used the OnTitleChange to display the page title in an Edit at the top of the TabSheet, as well as in the log (to be able to follow the action)

• we implemented some kind of "access token cache". We did not want to hardcode the access token in the tEdit .DFM. Therefore, whenever we request and get a new access token, we save it in a file, and this file is then loaded at the start of the next execution. This avoids to hardcode the access token (which changes after a couple of hours).

• to finalize our Indy version, StackOverflow, as usual, was our best friend, and Remy LEBEAU posts helped a lot.
  

7.3 - Oauth2

• I wanted to understand Oauth2 because a customer started talking about "Rest Services Security". At that time, I figured that Oauth2 would be an improved version of Oauth1. I now understand that this seems not to be the case, and the creator of Oauth2 had jumped ship.

  Anyway, the customer decided to use HTTPS and this saved me the trouble to create a Oauth2 SERVER.

• for simplicity, we chose to develop the whole thing on the same PC. DropBox allows, "for debugging", the use of a "localhost" redirection URL. This is the only case where HTTP is accepted. For production, the redirection URL MUST be HTTPs.

  But the problem is more serious: being at the same time the OWNER and the CLIENT on the same machine somehow obscures the working of the OAuth2 protocol : who is doing what and where.
  

7.4 - Delphi Components

• if you seriously want to implement DropBox, you will find a couple of Delphi Components using Google.

  TMS also sells a Cloud Pack which includes (along with Google Calendar, Mail, Contacts etc, Microsoft One Drive, Live Calendars, Outlook Mail, FaceBook, Twitter, Linkedin, PayPal) a DropBox Rest Client

• since I mentioned SSL, I had to create Auto Signed certificates. Using the command line (or a Delphi program with CreateProcess) was quite tedious. I then came across the ICS SSL upgrade. François PIETTE and his team have done a wonderful job there with 22 demos, including the OverbyteIcsPemTool.dpr which creates the self signed certificate with one clic. Of course, for production we will use Let's Encrypt free certificates, or rather some paying certificates.

  But among the 22 demos, you have HTTPS, JOSE ( Json Object Signing), POP3 and SMTP SSL, FTP SSL etc.

• aside from Delphi and ICS, there are also MANY REST Frameworks available for Delphi: Mars Curiosity, Delphi MVC, Mormot, Habari etc. So far I located over 12 of those ...
  

8 - Download the Sources

• zip_delphi_dropbox_rest_client.zip: the Delphi DropBox Rest Client, using the tRestClient components (156 K)

• zip_delphi_dropbox_indy_rest_client.zip: the Delphi DropBox Rest Client, using Indy Http Client with SSL (162 K)
  

The .ZIP file(s) contain:

• the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any other auxiliary form
  
• any .TXT for parameters, samples, test data
  
• all units (.PAS) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• for Delphi 6 projects, can be used from any folder (the pathes are RELATIVE)
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
  

• create or select any folder of your choice
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.
The .ZIP file(s) contain:

• the main program (.DPROJ, .DPR, .RES), the main form (.PAS, .ASPX), and any other auxiliary form or files
  
• any .TXT for parameters, samples, test data
  
• all units (.PAS .ASPX and other) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path outside from the container path creation etc).
  

• create or select any folder of your choice.
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.

As usual:

• please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  
• we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  
• or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button

  Name :
  E-mail :
  Comments * :

• and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.
  

9 - References and Links

• Rest Services Directory find the documentation about more than 20.000 web services

• DropBox:
  
  • DropBox V2 documentation
    
  • OAuth Guide with this diagram:

    

  • in the DropBox API page, the examples use the CURL. This is a (unix) command line tool allowing to sent HTTP (and other) requests
    
    
• Delphi did include DropBox in the RestDemo sample. This sample is included in the Delphi distribution, or available on SourceForge

• concerning Rest Web Services, we must pay our tribute to Marco CANTU who was the first to promote REST into the Delphi World. I still have articles dating back to 2009 (I archive everything, fearing the site might vanish). Still interesting is REST in Delphi 2010. Plus the many other posts, webminars, conferences and Youtube videos

• TMS also sells a Cloud Pack which includes (along with Google Calendar, Mail, Contacts etc, Microsoft One Drive, Live Calendars, Outlook Mail, FaceBook, Twitter, Linkedin, PayPal) a DropBox Rest Client

• FAQ Using tRestOAuth : the OAuth faq of Overbyte with the many OpenSSL additions
  

• we are doing Delphi custom developments. Right now we are finishing a Rest Server and a test Rest Client for a customer to export invoices and purchase orders.
  
• Delphi oAuth2 Tutorial Video : Jim McKeeth asked us to produce a 10 minute video for CodeRage 2019. We chose the oAuth2 subject. This video was constrainted by Embardacero to less than 10 minutes, so it is quite short, and in my optinion, is less detailed than this paper. But for those who prefer the image to the text ...
  

10 - The author