Examples/Demo_6_celery_aws_batch

[svnisha]

During the execution of this example demo_6 , the job are schduled in batch and completed successfully..
but iam confused to understand in the documentation, what is the intension of this example and what are we expected to see the results as and how.

secondly, the jobs in aws batch - how to restrict the number of jobs only to be executed in batch , the (range)
during my execution i saw the jobs were executing recurrisevly till 155 and i had to stop abruplty.
as , i saw the instances were scaling automatically to full fill the jobs horizontally and vertically with c4.large,c4xlarge and m4.2xlarge...

[vidyaWS]

Thanks for the comment - it is great to hear that the Batch example worked. This demo shows how to run jobs on Batch, and the results can be viewed on the Gremlin graph. The intention behind this is to replace the code inside the functions (components) with the compute invocation (simulation and ML, etc.) code which calls/invokes the containerized code in the ECR container. For example, this could be a digital twin simulation - we can work together to set this up. The idea behind TwinGraph here is to run multiple containers simultaneously asynchronously (as you have, with autoscaling) as well as chaining multiple containers together in an event-driven workflow chain (one container provides inputs to another once its completed successfully). We can walk you through more meaningful examples.

Secondly, this number is specified within the pipeline section of the deployment.py code - it is set to 372 (250 and 122) in lines 23 and 28 - you can change this as you wish. These tasks are very short arithmetic operations so it should not incur much cost. You can setup instances accordingly in the compute environment of Batch. When you create the Batch compute environment, selecting 'optimal' will pick those c, m, etc. instances, but you can also select specific lower cost ones such as t instances instead of optimal.

Please let us know if you have other questions! Thanks very much.

[svnisha]

Thank you , Vidhasagar for ur swift response !!

Demo_4_ Celery_backend : iam getting this error :
ERROR:celery.worker.consumer.consumer:consumer: Cannot connect to redis://localhost:6379/1: Error 111 connecting to localhost:6379. Connection refused.. Trying again in 14.00 seconds... (7/100)

Previous examples of demo 1 and 2 are success. Kindly let me know if anything missing .

aside, in the examples demo_8_ : i understand each function is run on its own underlying service and we chain together through a pipeline.

@pipeline(lambda_pipeline=True, batch_pipeline=True, Kubernetes_pipeline=True, celery_pipeline=True, celery_concurrency_threads=16, redirect_logging=True)

Query ist a need to create lambda_pipeline , batch_pipeline ,Kubernetes_pipeline ...

[vidyaWS]

Could you kindly check if the docker container for redis is running? There is a script provided with the latest commit (if you do git fetch & git pull in the directory you would be able to get it) called 'start_containers.sh' which you can invoke with 'sh start_containers.sh'. I suspect this could be the problem.

For demo 8 this is only for cost and compute grid placement optimization - there is no need to run all of these, but the demo shows you how to easily do so with twingraph if the architecture requires it. For example, you may need to run many queries which can be cheaper on Lambda, or the code may already be running on Kubernetes, so parts of the pipeline can be delegated there. As for the 'batch_pipeline', 'kubernetes_pipeline', etc. these are not separate pipelines, it simply tells TwinGraph that the pipeline includes these features. Please let me know if this answers the question!

[svnisha]

Thank you !!

Will build my own demo as suggested with some aws services and will reach to you & satheesh for help..