Redshift
AWS Data Pipeline
AWS Data Pipeline is IAAS (infrastructure-as-a-service), data pipeline is a web service, it supports data transformation in AWS storage like Amazon RDS, Amazon DynamoDB, Amazon Redshift.
Data Pipeline can be scheduled, and it will get run on schedule time, also it will run in frequency (A schedule can be executed every 5 minutes or 1 hour, every at once, or weekly or monthly or yearly).
AWS Data Pipeline you create, update and delete using the console or AWS CLI
Creation:
aws datapipeline create-pipeline –name TEST_PROCESS –unique-id TEST_PROCESS –tags key=ApplicationGroup,value=Test key=CostCenter,value=Test key=TEST,value=C.IT.1234567.1234 key=FundingSource,value=Test key=Test,value=1234 key=AIN,value=999 key=Landscape,value=TEST
Pipeline definition
aws s3 ls s3://test-t/datapipeline/test_service/jsonpath/test_process.json
Activate new datapipeline
aws datapipeline activate-pipeline –pipeline-id df-00123456ASDFGLKJHGAS
Deactivate existing datapipeline
aws datapipeline deactivate-pipeline –pipeline-id df-00123456ASDFGLKJHGAS
Delete:
Console:
List the Pipelines > Select the pipeline > Click Actions > Delete > Confirmation Click Delete.
CLI:
aws datapipeline delete-pipeline –pipeline-id df-00123456ASDFGLKJHGAS
How to create update and delete the AWS Lambda function?
AWS Lambda function creation using AWS CLI or AMW Console.
Lambda Creation:
aws lambda create-function –function-name TEST_PROCESS –runtime java8 –role arn:aws:iam::123456789010:role/TestExecution –handler com.rrmaha.lambda.TestServicePipeline –zip-file fileb://TestLambdas.jar –memory-size 512 –timeout 300
Lambda Update:
aws lambda update-function-code –function-name TEST_PROCESS –zip-file fileb://TEST_PROCESS.jar
Lambda Backup:
aws lambda get-function –function-name TEST_PROCESS > TEST_PROCESS.jar
Lambda Delete:
aws lambda delete-function –function-name TEST_PROCESS
Docker
This post we are going to learn about Docker advantages and disadvantages.
Docker is an open source tool, which can use for application deployment using lightweight containers, applications will work efficiently in different environments. Mainly used for the Agile-based project.
Advantages:
- Docker containers support CI/CD implementation
- Docker containers occupy less space
- Fast boot-up time
- Containers have a faster and better performance
- Easy to scale up
- High efficiency
- Portal to across different platforms, Windows or Linux or Cloud
Disadvantages:
- Connect to existing load balancing services/database
- Docker is platform dependent
- It has poor monitoring
Microservices
This post we are going to learn about Microservices advantages and disadvantages.
Microservices is from service-oriented architecture, smaller services instead of one big application, the application will build as many smaller services, applications will run their own, also it can be using different coding/programming languages.
Microservices frameworks for Java – Spring Boot, Jersey, Swagger.
Advantages:
- Replace the monolith application
- Microservices applications are very small and easy to handle
- Container-based application, easy to develop, faster development and deploy the services
- Each service can be developed in different languages
- Microservices we can use it for continuous delivery
- Easy to learn and lightweight
Disadvantages:
- it will lead to too many services
- Network latency
- Too many processes and calls
- Testing will be bit complicated
Python
This post we are going to learn about Python advantages and disadvantages.
Python is an interpreted high-level programming language, Python was first released in 1991, developed by Guido van Rossum. Python is used in a range of applications like dev ops, server-side scripting, web development, scientific computing, and education.
Advantages:
- Open source
- Easy to learn
- Less code with faster development
- Scalable and Extendable
- Object-oriented
- It supports multiple platforms and operating systems.
- It supports standard library
Disadvantages:
- Python is slow
- Python is not good for mobile computing/development
- Python is not good for game development
- It has a limitation with Database access
- Python is not good for memory intensive tasks
- Lack of multiprocessor support
- Concurrency and parallelism are not designed
Bootstrap (Front-end framework)
This post we are going to learn about Bootstrap advantages and disadvantages.
Bootstrap is an open-source front-end framework use to develop the responsive websites and web applications.
This framework is built on using HTML, CSS and JavaScript.
Advantages:
- Lightweight framework and open source
- Responsive design framework
- Supports for all the major browsers (Chrome, Firefox, IE, Opera and Safari)
- Good grid system
Disadvantages:
- It required a lot of CSS overwrite
- It uses jQuery
- The sometime problem with big display/screen
DevOps
This post we are going to learn about DevOps advantages and disadvantages.
DevOps (Software Development and Operations) is a set of processes that automate the build, deploy, test and release (Development, QA and Operations).
Advantages:
- Increase the service quality
- Continuous software delivery
- Operations will be more efficient
- Faster resolution of the problems
- Stable operating environments
- Improved communication, collaboration and customer experience
Disadvantages:
- DevOps have limited transparency
- Lack of domain knowledge
- Many of the developers do not get access and they do not get much visibility in DevOps process including production environments server details.
- Not suitable for legacy code
- Require additional tools and hardware, which can add more operating cost.