Rohan Lopes Blog

Data engineering at high level is to design, build, monitor and secure the data processing pipelines.  Data pipelines mainly perform ETL (Extract, Transform, Load) jobs, which transform the source data (structure/un-structure) to a data warehouse in a meaningful way, which can than be used for analytics or machine learning models.  Data processing can be in form of batch or streaming. On GCP, we can use Dataproc (Apache Hadoop, Spark etc.) for Batch processing and Dataflow (Apache Beam programing) for Batch & Streaming.  Batch Pipeline: Processing the data in the form of batches. Examples: A nightly Job to extract sales transactions for analysis.  Streaming  Pipeline: Processing continues stream of data. Examples: IoT events, payment processing events, logs. Streaming processing is used when you need a near real-time outcome or decision making.  To get started on Data engineering, would recommend you to go through below courses or labs.  Modernizin...

  Google Apigee X is the API management platform which allow to develop and manager experience API's.  This APIs can be made available to multiple clients with different security policies.  Example, a Mobile Developer might need a JWT access token validation policy vs a traditional system might need same API with HTTP Basic policy.  As an API management platform, it allow several security policies like SQL injections, JSON Threat Protection etc.  Google Apigee X is full managed API management platform. It consist of a management plane and runtime plane. You can also implement Hybrid model i.e. Cloud & On-Premise runtime instances.  You can refer, below link for high level architecture: https://cloud.google.com/apigee/docs/api-platform/architecture/overview For a hands on experience and details overview, would recommend completing below course and labs: Course: API Design and Fundamentals of Google Cloud's Apigee API Platform       ...

Consider a scenario, where you as a API developer been ask to to implement the API's for a new mobile/web application team. The requirement is to implement a application to allow users to manage orders.         Mobile APP -> API infra->Backend enterprise system While implement a full working API might take days, you also do not want the application team to wait till the API's are ready. Thus, in most case we end up implementing the mock API's with sample scenarios. In this post, we will define an API using swagger openapi 3.0 yaml specification in swaggerhub. We will then use the definition to implement mock API's in postman. Even though postman provide option to define the API's, i liked swagger hub much better, it allow you to view your API on right side.  For this post I implemented API definition in swagger, please find the below link. https://app.swaggerhub.com/apis-docs/lopesrohan1988/ordermanagement/1.0.0 API Definition can be found at below link: h...

As part of this article we will learn below: Create the cloud storage bucket Use sample hierarchical JSON payload to auto create bigquery table Create google cloud function to listen to the object creation event on storage bucket and Fetch the JSON payload from the file and load in to Bigquery *We will be using Python for this activity. 1. Create the cloud storage bucket This is the simple task, you can either create the bucket using the console or gsutil command. For this exercise, we will use console. We will ignore some parameters like regional, bucket type, as we will not be using this bucket for large storage. Go to cloud storage -> Click on Create Bucket Give some unique name, you can use your Google Project ID+some name and click on create bucket. 2. Use sample hierarchical JSON payload to auto create bigquery table One of the quick way to create the Bigquery table to save the JSON payload with array is to first create the sample JSON payload and use it to create the bigq...

In the exercise we are going to use the Kaggle cats and dogs data. Some of the code is from the training course "TensorFlow in Practice". Part1: First step towards building the machine learning model is to prepare the dataset. In this notebook we will perform below: Download the kaggle cat and dog data set Extract the zip Upload the data set to Amazon S3 bucket #Download the kaggle data set ! wget --no-check-certificate \ https : // storage . googleapis . com / mledu - datasets / cats_and_dogs_filtered . zip \ - O ./ cats_and_dogs_filtered . zip Extract the zip file to local directory import os import zipfile local_zip = './cats_and_dogs_filtered.zip' zip_ref = zipfile . ZipFile ( local_zip , 'r' ) zip_ref . extractall ( './Data' ) zip_ref . close () Use Amazon SDK to upload the data to S3 bucket . I created the bucket name "sagemaker-05may2020842" #Copy the data to AWS from Loc...