MLCC


Let me share key take away points from meetup event "Google Machine Learning Study Jam

In Feb/March 2018, Google announced MLCC Machine Learning crash course In July 2018, MLCC Study Jam series comes to India. Click here and click here to know more. I attended one such event with my friend, by Industry 5.0 meetup.

Here are few useful links

TensorFlow Content Bundle, Spring 2018

Gradients and Partial Derivatives : YouTube Video 
Later on, I found the Maths play list is good. All videos  by  Eugene Khutoryansky are excellent

Another YouTube video by  Christopher Gondek. Here also, the playlist about 'Machine Learning Visualization' is good. 

Microsoft announced about FPGA based Edge Computing : Brainwave project

AutoML and transfer learning. At present, they are at nascent stage. Once let it fully evolved then we may not need people who know AI/ML. The machine themselves will learn. I did little Googling and found few links : http://www.ml4aad.org/automl/ and https://automl.info/

As per my knowledge, after completing any Machine Learning course till one completely switch his/her career path, Kaggle is the only platform to get hands-on experience. I came to know one more such platform Seedbank  I found one seed about 'Piano Transcription' quite interesting. We discussed with Sanjay Chitnis about creating similar seed to recognize Indian Raga

There is an interesting book 'Pattern Recognition and Machine Learning (Information Science and Statistics)' by Chrisopher M. Bishop

http://playground.tensorflow.org is an excellent, browser based Neural network tool. It is also used as part of MLCC We discussed about L1 regularization, L2 regularization, confusion matrix, precision, accuracy, recall, F1 score, Receiver operating characteristic etc. Precision is all about how many positive case, the algorithm could able to detect out of all positive cases. Recall is about how good is the diagnostic test? 

CNN is combination of filter and dimension reduction. RNN is a special case of LSTM. GAN is widely used to creation. The GAN Zoo has list of all variations of GAN

We also discussed about Semi-Supervised Learning , Topic Learning OR Keyword Learning, that is beyond supervised learning, Gold standard etc. 

At the end, Sanjay drew out attention to an interesting trend that now, product cost is keep reducing. Features in products are keep increasing. Service cost is keep increasing ! India has lots of data available. There is good scope of data analytics and machine learning for General Election 2019 at India. 

DevOps & Digital Transformation


DevOn (Prowernesss) conducted its first meetup with core theme of "DevOps and Digital Transformation". We noticed many buzzwords in Software Quality. It started with CMMI, then P-CMMI, then Agile, Scrum and now DevOps. The most important is, how any orgnisation tunes the latest trend with (like DevOps) with its business strategy, culture and customer expectation. Uber became successful with this approach. 

Hariharan Ganesan broke myths that Rolls-Royce is no longer making cars ! He himself does not posses any car by Rolls-Royce and neither he is getting any discount to buy one. :-) He talked more about Rolls-Royce, its business, product, strategy, market, people etc. In spite of being company based on metal, nut and bolts, DevOps is relevant for Rolls-Royce. 

We were amazed by less-known facts about aviation industry. The first IoT enable business was aircraft. Its engine is not sold, but offered on rental per hour basis. It is an intelligent engine and having engine to engine communication capability. Rolls Royce outsources 10mm size small robot (size of a bug) which goes inside of aircraft and takes inside engine photos. R2 (R square) Data Labs is sister concern of Rolls Royce. Key criteria are (1) availability : avoid flight delay and flight cancellation (2) safety and (3) fuel efficiency. At R2 lab, there is a 'digital twin' of each engine. Digital Twin is much more than Aut-CAD or CAM. It is mathematical model (relatively simple) and/or physical model based on CAD/CAM running on HPC. When aircraft is flying over Atlantic ocean, its engine talked with other engine about yesterday's weather condition over Atlantic ocean to compare. At R2 Labs, weather data is fedded to digital twin of the engine for to fine tune and make better engine performance 

Prashant Kumar talked about Docker and Kubernetes. As per definition : docker is combination of code, runtime, system tools, system libs etc. It is most popular container. Docker image is immutable. DockerHub is docker registry - something like git, github, big-bucket. Docker is popular container. Docker improves DevOps workflow by (1) environment consistency (2) Isolation (3) organizing applications and (4) portability. One must mention/use correction version of docker engine for portability. Prashant gave tips about 'dangling API' to clean manually all the unused objects, else the file system will be out of space. Kubernetes maintains containers at scale. A pod can have one or more than one containers. Sometimes, for state-full services we need to attached volume like EBS. 

Vinay Krishna talked about ' DevOps Success Recipe: One Team One Goal ' The process and tools are just means. They may not solve actual problem. Vinay explained few real life scenarios, without mentioning any organisation name. 

1. Developers re-designed a web page that was used by customer support team, without informing them. The team faced lots of issue while answering the customer call and at mid night the software needed to be rollback to previous version. 

2. The end-users were involved from beginning. Developers completed many modules, and then started GUI. The end-users raise concern about GUI and finally GUI was re-designed after many e-mail exchanges (to blame) . Involving the end-user may not help, if they attend the meeting just for shake of attending. 

3. All stack holders were involved. Code coverage was 80 percentage and above. CI/CD pipeline was green. All test cases passed. There was 100 % automation. No severity three bugs, yet customer was not happy. All bugs were moved as severity.  Many test cases for code coverage was not up to the mark and few does not have even needed assert statement. 

Yes all three amigos (as per agile) business, developers and testers must be involved as one team. Once someone commented that this task belongs to new DevOps team. There were already three teams, (1) Developers, (2) QA (3) Operation. DevOps team members were same people from operation team and after Docker + cloud training the team was re-branded as DevOps team! Operation people does not like development work. Developers does not like to do work , that is done by operation team. All needs to work as one team. 

4. Prashant also shared a positive example about valuable suggestions from operation team like : (1) Add human readable logs (2) add feature switch turn on/off etc. 

Prashant also talked about NetFlix tools like Chaos Monkey, Simian Army, Chaos Gorilla etc. They are resiliency tools that helps applications tolerate random instance failures on cloud. He suggested to read "Value Stream Mapping" and "Software Horror Stories"

Now let me share my views. This talk remind me about similar practical facts that I learnt during MBA. When SPC (statistical process control) introduced, the shop-floor people does not understand about 2 lines in chart recorder. They drew manually to keep chart withing two lines !! It reminded me the famous business novel 'The Goal' about ToC (theory of constraints). In the novel the people strongly rejected reporting so many numbers/statistics. Later on, new system of using green tag and rad tag itself caused another issue.  Now time changed. I also remember a famous joke when a US firm ordered something from Japan with 97% quality, the Japan team prepared 3% pieces separately with poor quality. 

In IT industry also, I observed that (1) sometimes, all the review process does not apply to 'system engineers team' even the clarity in requirement is the most crucial part. (2) Sometimes, quality people have un-realistic matrices about number comments within kLOC, to meet them, manager add dummy comments or remove genuine comments from the record. It defeats its purpose. 

So in summary, the human tendency, the culture, power politics cannot be ignored to bring the change. Quality must be everyone's responsibility. 

Shynish Meladath talked about Industrial IOT and DevOps. The key take away points were 

1. 'Particle Photon' It is an IoT kit 
2. Eclipse 4diac  provides an open source infrastructure for distributed industrial process measurement and control systems based on the IEC 61499 standard.
3. resion.io that brings Linux containers to IoT. 
4. Kaa is an open source IoT platform
5. Spanner CI is continuous integration for IoT
6. macchina is a versatile platform for car. 

I missed the last session "Data-Driven DevOps" by Ashwin Shankarananda, due to other priority. 

Disclaimer: I did my best to capture notes and key take away points from the event. However the content is as per my understanding and it may or may not reflect the original content/intention by the speaker. Any corrections are welcome. 

Reference https://www.meetup.com/Digital-DevOps-Bangalore/events/252154106/


OpenStack meetup


16th June 2018, I attended OpenStack Meetup at Ericsson office. Let me share my notes for readers of this blog : Express YourSelf !

Shashi Singh from Altiostar Networks discussed about EPA (Enhanced Platform Awareness). 

EPA is about about making aware NFVO, VNFM and VIM, that specialized hardware is available below virtualization layer. E.g. High I/O throughput, high performance CPU, GPU, crupto accelerators and many more as below slides:







Shashi explained nicely NFV MANO architecture to build the context and introducing the acronmys. Telco NFVI providers are: RedHat, WindRiver, VMWare, Mirantis etc. VNFM is categorized as specific VNFM and Generic VNFM. It supports three interfaces: Ve-Vnfm-vnf, Vi-Vnfm and Or-Vnfm.

He explained how EPA can eliminate the need of passing through virtualization layer for data packet, if the required VNFs are running at same CPU socket. I confirmed my understanding that, one example of EPA is let all VNFs for user-plane data having single CPU afinity. We also discussed about SR-IOV single root input/output virtualization, cpu pinning, threading policy etc. Sometimes within storage node, one can leaverage use of SR-IOV, DPDK etc to support more I/O. TOSCA standard defines combination of NS-D (Network Service Descriptor) and VNFD (VNF Descriptor). Shashi also mentioend about Queens Release, Cyborg framework, nova, ironic etc. 

Here is list of Intel technologies for EPA

1. Intel Advance Encryption Standard - New Instructions (Intel AES-NI)
2. Intel Advance Vector Extensions (AVE) and AVE2
3. Intel Quick Sync Video Technology
4. Intel QuickAssist Technology for encryption / decryption and  compression / decompression
5. Intel Trusted Execution Technology (TXT)
6. Intel Node Manager : Server Mangement at Data Center
7. Data Plane Development Kit (DPDK) at Xenon processor
8. SR-IOV
9. Intel Xeon Phi Co-processor: for PCI

I came to know about this website https://www.telecomtv.com/ During tea-break, someone commented, that Kubernetes is now open source, but it is very old. Google is working on new technology / product named by Omega that is yet to be open sourced. 

Palaniswamy from Tech M, explains about ManageIQ (with demostration) as Multi Cloud Management Platform. ManageIQ supports public clouds like : Amazon Web Services, Microsoft Azure, Google Cloud Platform; OpenStack based private clouds; containers like Kubernetes, OpenShift Origin etc. ManageIQ internally uses PostgreSQL DB. Ansible Tower is used for configuration and automation. 




Sukant J R and Manoranjan Sahoo from Ericsson presented about CI/CD for containerized openstack development based on Helm. 




We also discussed about 4 types of people in IT industry always remains. (1) Developers (2) Support engineers (3) Integrators and (4) Testers. The new technology comes and goes. One needs to work, as per his/her core strength. 

Apart from that, Uday T.Kumar from Ericsson shared some insights about OpenStack Summit and how to contribute to OpenStack community. He also acknowledged that Bangalore OpenStack community is very active and sharing the latest updates. Later on those updates are known to entire world at OpenStack summit. 

Disclaimer: I captured this notes, as per my understanding on best effor basis. So it may not accurately refelct the spearker's view. Any corrections are welcome.  

Reference: 
https://www.meetup.com/Indian-OpenStack-User-Group/events/249891291/
https://01.org/sites/default/files/page/openstack-epa_wp_fin.pdf
https://networkbuilders.intel.com/network-technologies/enhancedplatformawareness

My Holiday Destination



HDH Mahant Swami Maharaj @ Bangalore


Jai SwamiNarayan

Recently, HDH Mahant Swami Maharaj visited Bangalore. All followers of BAPS SwamiNarayan religion were super excited with full of devotion. If one remains in the presence of such divine Saint, even for short time, he/she get huge benefits in all fronts. I was lucky enough to get His blessings and listen to His speeches. He emphasize on importance of unity among all followers/volunteers for great success. 

One of BAPS followers planned to sing Kirtan/Bhajan/rhyme in different languages : Hindi, Panjabi, Gujarati and Sanskrit. I just contributed by translating one such Gujarati hymn

ગુરુદેવના ચરણમાં કરીયે કરોડો વંદન 

into Sanskrit. Here it goes for the benefits of all readers of this blog "Express YourSelf !





गुरुदेवनाम् पादयोः।           कुर्महे कोटि प्रणामान्
परब्रह्मन: पादयोः           कुर्महे कोटि प्रणामान्

भवसागरम् तारयन्त:           न: नाविका: भवन्त:
दीनबंधव: भो: रसवन्त:           कुर्महे कोटि प्रणामान्

वयम् दीन हीना: आगता:           न किमपि नीता:
भवद्भि:  ह्रदा  स्विकृता:            कुर्महे कोटि प्रणामान्


भो: श्री हरिसखीन:           श्रीजी सदृशा: प्रतापिन:
अन्तरे वसन्त: शाक्षिन :            कुर्महे कोटि प्रणामान्

भवन्त: अमृतम्  वर्षन्त:           तथापि वयम् पिपीषन्त:
परम् ह्रदया: रमन्त:            कुर्महे कोटि प्रणामान्

सदा नेत्रेषु वसन्तु              शरणे न: स्थापयन्तु 
वयम् पादयोः वसेम            कुर्महे कोटि प्रणामान्

Translation: 

we all are bowing down at lotus feets of the master for crores of times. 
he is like supreme Bhraman. We all are bowing down at lotus feet of supreme Bhraman for crores of times

oh master, you are making us to cross this worldly ocean (Bhava-saagar). you are like Helmsman /sailor /caption of our boats. 
you are very interesting and relative of all poor people. we all are bowing down for crores of times. 

we are poor and with less reources. We did not bring anything. 
(in spite of that) you accepted us with full heart. we all are bowing down for crores of times. 

you are close friend of God. You are as powerful as God.
you are like witness staying in our hearts. we all are bowing down for crores of times. 

you did rain of nector. in spite of that we are thirsty. 
but our hearts are feeling pleasant. we all are bowing down for crores of times. 

please stay always in our eyes (we always want to see you everywhere). we all surrender you, please accept us. 
we all wish to stay at your lotus feet. we all are bowing down for crores of times.

Alexa


I attended few sessions by Amazon about its Alexa devices, during last few months. Voice represents next major disruption in computing. These devices provides VUI (Voice User Interface). We had hands-on sessions and interactive QAs. Let me cover some of the relevant URLs and overview in this blog post for readers of "Express YourSelf !"

Comparison 


Here is a quick comparison of all major devices:



Echo Dot Echo Dot
Kids Edition
Echo Echo Plus Echo Spot Echo Show
Price Rs. 4000 N/A Rs. 10,000 Rs. 15000 Rs. 13000 N/A
$50 $80 $100 $150 $130 $230
Rs. 3341 Rs. 5346 Rs 6682 Rs. 10023 Rs.8687 Rs. 15369
Microphones 7 7 7 7 4 8
Misc. Smart hub 2.5" screen 7" screen

Apart from the regular devices from Amazon, few smart cars and smart TVs also have built-in Alexa support. Amazon also launched "Alexa 7-Mic Far-Field Dev Kit" that hardware can be part of any product. One can add display support also like Echo Spot and Echo Show, however it needs to go through rigorous certification process from Amazon.

Comparison of Mobile App with Alexa Skill

Mobile App ~ Alexa Skill
Mobile App icon ~ Invocation Name
GUI ~ VUI

Many mobile apps have Alexa skill e.g ola, goibibo, crickinfo, zomato etc. have alexa skill


How it works

Alexa software has mainly two major components

1. ASK (Alexa skills kit) to build new skill

2. AVS (Alexa voice service) to integrate with RPi kind of device. 

The hardware is quite simple with microphone array and speaker. The microphone array used for noise cancellation. The spoken sentence is divided into :

1. Wakeup word
2. launch
3. Invocation name. It should be two words. 
3. Utterance

Steps

step 1. Wake up word can be = Alexa / Computer / Echo / Amazon

This will wakeup the device. It triggered beam forming to listen. 

step 2. The utterance (captured audio) goes to cloud

step 3. At cloud real magic happens with

3.1 speech processing 
3.2 NLP


step 4. The invocation name  is detected. With invocation name, the execution flow goes to specific skill. Now skill has all the logic, algorithm to further understand the utterance, to access cloud service, database etc and finally for the response

Here the front-end is developed and tested with simulator using developer.amazon.com

step 5. As per training model, Alexa translate the utterance to Intent. The developer need to create custom intent, that mapped to function implementation to provide response. Alexa also provide standard built-in Intent, that developer can implement
https://developer.amazon.com/docs/custom-skills/standard-built-in-intents.html
There is a set of built-in intent libraries for various use cases
https://developer.amazon.com/docs/custom-skills/built-in-intent-library.html


In Alexa terms "slot" is like argument to function. Alexa has built-in slot types : https://developer.amazon.com/docs/custom-skills/slot-type-reference.html

There is many to one mapping between utterances and intent. There is one to one mapping between intent and function

There is many to one mapping between utterances and custom slot. There is one to one mapping between custom slot value and argument value to function. 
So one can pronounce "A.C" or "Air Conditioner" still it maps to same enumarated value as argument to function. Such synonymous are detected using "Entity Resolution" 

The back-end function can be implemented at any HTTPS terminated end-point or AWS lambda service. The AWS Lambda service, at present, is available only for regions: 
1. US east North vergina
2. EU (Ireland)

The professional skill can use session attribute for better user experience and also for data analytics. 

step 6. The response can be 
6.1 Speech : SSML, Local lingo, TTS, audio stream, small mp3 files
6.2 Cards = title, subtitle (skill name), text (content), image. 
Cards are optional. We can use rich text with different font including Unicode at card. It is built using various BodyTemplate and ListTemplate. 

The speech output goes to speaker. The card output goes to 
1. Alexa Companion App
2. Echo Spot and
3. Echo Show

One can check device capability for including card/video in response. 

The Alexa skill can be built using pre-built models

1. Custom: For unique need
2. Flash briefing : For RSS feed
3. Smart Home : For home automation
4. Video : For video application

Questions - Answers

Let me highlights few leanings about Alexa Echo eco-system and the devices

* The Alexa companion app can be connected to only one device. So it is not possible to push same image/content/card to all companion app running on mobile using single Alexa device
* Amazon allows to use same invocation word for multiple skill developed by same/different people. All such skill can be configured for given device. However the skill that is configured last, it will be invoked for the duplicate invocation word. 
* It is possible to enable/disable specific app on the device using mobile app
* It may possible to develop smart home device using Raspberry Pi for single user, with skill that is not published. One can use Smart Home pre-built model. Let the Intent invoke code running at Raspberry Pi, that turn on/off home appliances using GPIO pin and relay. 
* None of the Alexa devices has built-in battery. 
* "Alexa for Business" can have features like allowing access to very specific limited set of skills only. 
* Alexa does not have any adult content, so parental control is not needed. 
* One can change wakup word and replace "alexa". However still it will be female voice only. The Alexa devices do not support response in male voice. 
* Alexa device cannot be used for dictation or speech to text conversion. One can use AWS transcribe service https://aws.amazon.com/transcribe/ for the same.  
* One can develop (1) one shot dialogue (2) multi-turn dialogue skills
* To design multi-turn dialogue skills, one can use (1) graph UI or (2) frame UI. 
* Alexa can prompt for missing slot
* Amazon is coming up with Notification, that will be triggered by skill to Alexa device. However until the end-user ask to get notifications, the Alexa device will not start talking by itself to inform about notification. 

URLs

Now, let's have a look to important URLs

alexa.design/guide : Design of Voice Experience 

alexa.design/indiacheckin :Join the Amazon developer community & check in for the event in India. 

alexa.design/india : It has details about all meetup, hackathon, webinar, slack channel etc.

alexa.design/codecademy  and alexa.design/training : Online learning resources

alexa.design/factskill and bit.ly/2JWxlY9: Getting started with skill development

https://github.com/alexa : Alexa public sample code repository

https://developer.amazon.com/en-in/alexa-skills-kit Getting Started in India

Alexa response can be further enhanced at skill using

1. SSML. SSML is Speech Synthesis Markup Language. More details:

https://en.wikipedia.org/wiki/Speech_Synthesis_Markup_Language
https://www.w3.org/TR/speech-synthesis11/

Alexa specific SSML : https://developer.amazon.com/docs/custom-skills/speech-synthesis-markup-language-ssml-reference.html

2. Speechcon https://developer.amazon.com/docs/custom-skills/speechcon-reference-interjections-english-india.html

Reference

Sohan Maheshwar
@soganmageshwar

sohanm@amazon.com

https://www.slideshare.net/AmazonWebServices/alx326applying-alexas-natural-language-to-your-challenges
https://developer.amazon.com/blogs/alexa/post/5b0efd02-0ed0-42d5-b922-5ee594d30a38/new-alexa-skills-kit-template-build-an-alexa-decision-tree-skill

https://www.slideshare.net/AmazonWebServices/alexa-bootcamp-skill-building-101
https://www.slideshare.net/RickWargo/how-to-build-an-alexa-skill
https://www.slideshare.net/AmazonWebServices/wks403-build-an-alexa-skill-using-aws-lambda-78928075
https://medium.com/tribalscale/getting-started-with-voice-faqs-from-our-partnersby-tribalscales-voice-experts-fd5edec6f042
http://www.smith.co/blog/2017/CAFFEINATED-COMMERCE---EPISODE-3-TESTING-CONVERSATIONAL-USER-INTERFACES

Distilled Python


Sometime back, I came across an excellent book on Python: "Fluent Python" (O'Reilly, 2015) by Luciano Ramalho. On Saturday, 28th April, I got opportunity to listen to him live at Geeknight workshop "Distilled Python: Features you must know to use it well" at Thoughtworks, Kormangala, Bangalore. Here is my note about the event, exclusive for readers of this blog: Express YourSelf !

He programms in Python since 1998. His speaking record includes PyCon US, OSCON, OSCON-EU, PythonBrasil, RuPy and an ACM Webinar. All the slidedecks are available at :
https://speakerdeck.com/ramalho



Generators and Iteration 

Generator allows lazy data processing. Here data is loaded to main memory, as and when needed. In Haskell programming language, almost everything is lazy data processing. At opposite, numpy Python module is about fast data processing. Here all data is loaded in memory for vector arithmetic. Intel introduced new instructions MMX (Multimedia eXtension) in mid 1990s. 

import sys
for arg in sys.argv:

print arg

Here is list of Python's built-in iterable objects and items they yield. 

  • str : unicode char
  • bytes: int 0 to 255
  • tuple : individual fields
  • dict: keys
  • set: elements
  • io.TextIOWrapper: Unicode lines
  • models.query.QuerySet : DB rows
  • numpy.ndarray : Multidimensional array , elements, rows 
Here are few use cases of Iterator in Python

Parallel Assignment is possible with iterable objects. 

It is also called tuple unpacking. However it is not specific to tuple. Here right side of = sign is iterable. 

pairs = [('a', 10), ('B', 20)]

for label, size in pairs: 
print(lable, '->', size)

Multiple values can be passed to function

This is also called star arguments. 

t = (3,4,5)
fun(*t)
def fun(a, b, c):

Reduction functions:

We use "map-reduce" in Big Data. Python has support for "map-reduce" However such reduction functions server the purpose of "map-reduce" 
  • all : boolean 
  • any : boolean
  • max
  • min
  • sum
Reduction functions consume iterable and provide single result. 
Python has mapreduce. 

One can write more readable code, where multiple and conditions are replaced by all and multiple or conditions are replaced by any

Sorting

sort() function only for sorting list. sort() sorts list in place. 

sorted(): a built in function. it consumes any iterable. It has keyword argument for sorting key

Here one can pass even function as argument. Unlike other sorting library in C/C++, here the function is not for comparison. It is about to generate key. 

To write poems, one needed words who ends with same characters. Here is source code. 

sorted(L, key=lambda s:list(reversed(s)))

Now let's have a look in details about iterator . Python has built-in support for iterator design pattern. There are two types of objects (1) Iterable (2) Iterator. 

Like food is etable, a collections of objects are iterables. iterable has methods like  __iter__ 


The iterator has state. It has methods like  __next__

Please note, the next method is not part of iterable object, as it can be shared by multiple threads. 

StopIteration excpetion raised by next() method. 

in python "for loop" obtains iterator from iterable. Then it repeatedly invokes next() on iterator. 

Now something about generator. In Python, generator is like synonymous of iterator, and can be used interchangeably, but they both are different. In Python its syntax is same as normal function. The generator is also defined with "def" keyword like function. However only generator contains "yield" keyword somewhere in code. 

One should not invoke __iter__, __next__ method directly. Here, Python acts like framework. So developer will not invoke those methods, but let Python as a framework invokes them, as and when needed. The developer can create our own dender methods __next__, __iter__ etc. in object. next(g) is implemented in optimized way in C language. 

In generator the execution flow is frozen at "yield" keyword and it gets resume later. So it is synchronize progrmaming without call back. So generator is introduced in JavaScript also. Please note, here "yield" is not same as "return" in function. The generator cannot be reset. 

Built-in generators of Python 
  • enumerate : returns first is number which increments and second is as per the input
  • filter: Python2 returns list. in Python3 one can go over data that does not fit in memory, using "filter" generator. 
  • map
  • reversed
  • zip: consumes iterables and generate tuple. if one iterable is shorter than zip will stop at shortest without any exception. In Python2 zip generator returns tuples that can be passed to list() constructor. it can be passed to dict() also. 
Now let's see about Generator expression (genex) in Python

1. list comprehesnsion
it is inspired from "Set builder notation" in Maths and Haskell programming language. 

l = [ord(c) for c in s]

Here "ord" function gives ASCII value for given character. The output is always a list. 

2.
g = (ord(c) for c in s)
it returns generator with laziness. 

To understand more, please refer 

This project is no where link with ISIS terrorist group :-) . In this project, instead of writing complex for loop content, the generator expressions and generators are effectively used. This code is about database migration with many command line options in main function. It captures inputs from one DB and kept it in generator for lazy evaluation. The output generator is populated by processing data from that input generator

pytest module

The post lunch session focused on TDD (Test Driven Development). Using pytest module, we can have test cases (TCs) without class. In Java, JUnit framework requires class to write TCs. So JUnit, CppUnit etc. Unit test framework are not Pythonic way. 

pytest.raise provides context manager. It has its own entry and exit method. It can be used to lock/unlock shared resources and open/close the file. 

@pytest.fixture is more like meta-data programming. Here fixture function is passed as argument to test function. 

There are many plug-ins to generate fancy reports on top of pytest. 

Python Data Model


Python Data model is not about data science. The better name can be Python Object model. It is all about various dunder methods to support many built-in feature of Python as framework. These dunder method should be implemented at user-defined class. Such methods are like new and delete methods in C++. The dunder methods are not protected method, even pycharm IDE indicates as private/protected, by mistake.method with __ as prefix is private/protected. If __ is as prefix and suffix both, then such method is dunder method. 

1. In Python all object should have method for string representation. Python have two dunder methods repr and str. The str method is invoked by print() for string represntation of the object. The repr method is invoked for debugging the object. 

Bobby Woolf inspired to add repr method to Python data model. The reprlib is very useful module to implement repr dunder method for user-defined class. For example if we use reprlib.repr for our own vector class, then it will (1) remove infinite loop from collection member variable and (2) it will print first 10 members only 

2. collection should have length

3. The iterable object should have method iter

4. The iterator method should have method next

5. The eq method is called for == operator. 

6. The init method in Python is not constructor. It is inializaer. It does not allocate memory.

7. The getitem method is very useful for indexing and slicing. 

Let's look at genex in few dunder methods for Vector class. 

def __eq__(self, other):
    return all(a == b for a, v, in zip(self, other))

This method will incorrectly, return True, if both vectors have different length and initial members are identical. We can use izip in place of zip. However, the better solution is, first compare length. 

def __abs__(self, other):
     return math.sqrt(sum(x*X for x in self))

Here are use cases, when Python invokes these dunder methods


1. arithmetic and Boolean expressions : operator overloading
2. impicit conversion to str e.g.  print(x)
3. conversion to bool when used if, while, and, or, not
4. attribute access, including dynamic or virtual attributes
5. emulating collections: o[k], k in o, len(o)
6. Iteration : for, tuple unpacking, star arguments etc. 
7. Context managers - with blocks
8. meta programming: attribute descriptors, meta classes. 

Then we had nice discussion about implementing __rmul__ method to implement product of scalar and vector, where both arguments can be in any sequence. The use of returning "NotImplemented" to invoke rmul. Even in all standard Python 3.8 libraries also we may not get implementation of __rmul__ method for any class. 

Typecode

Type codeC TypePython TypeMinimum size in bytes
'c'charcharacter1
'b'signed charint1
'B'unsigned charint1
'u'Py_UNICODEUnicode character2 (see note)
'h'signed shortint2
'H'unsigned shortint2
'i'signed intint2
'I'unsigned intlong2
'l'signed longint4
'L'unsigned longlong4
'f'floatfloat4
'd'doublefloat8

Miscellaneous 

coroutines is another nice Python feature. We can use keyword async along with coroutines. As per David Beazley's advice: coroutines are not for generators. 

We should use exact same error message as Python reports, in our custom class, so one use the error message in stack overflow searching :-)

fractions.Fractions is vary useful module, who stores numerator and denominator separately.  

Head First Design Patterns is another book similar to GoF Design Patterns 

Python is easy to use and very popular so investing your time and efforts in Python learning, gives fast returns. 

Jaydeep - the event organizer stressed upon, various plugins for pytest module, to generate test automation fancy reports for people at different hierarchy. Here is one such module at his github repository :  https://github.com/jaydeepc/report-mine

About various programming languages

Go and Python: Both progrmmaing languages allow to write code without using class. On other hand, in Java Maths class has only static methods, yet class is needed. 

Python understands iteration, better than C. In C, programming, index variable i is needed. It is not needed in Python since 1991. Since 2004, Java also does not need i. This is borrowed from CLU language by Barbara Liskov. CLU language was not commercially successful but it influenced many programming languages. C does not have iterable object. Go : limited set of iterable objects. One cannot create iterable objects in Go language. :-(

"0" is true in Python. It is true in C also. As it contains a string with '0' = 0x30 character. However "0" is false in JavaScript

In other languages, exception indicates abnormal error condition. While in Python to raise signal also, exception is used. So the generator are introduced in JavaScript also. 

Object Oriented Programming are design patterns for non-OOP languages. As we know, Iterator is design pattern for OOP languages, except Python. Python has built-in support for iterator design pattern. 

In Python the number overflow never happen, unlike other programming languages. The variable is automatically promoted to data type with next higher level of memory allocated. 

The Python module "itertools" is inspired by Haskell programming language. If you have not used "itertools" module, then most likely you might have written code, that was unnecessary. Few example of itertools: 
  • infinite generators
count(), cycle(), repeat()
  • generators that consume multiple iterables
chain(), tee(), izip(), imap(), product(), compress()
  • generators that filter or bundle items
compress(), dropwhile(), groupby(), ifilter(), islice()
  • generators that rearrange items
product(), permutations(), combinations()

"I have a problem. So let me use 'regular expression'."
"Now you will have two problems" :-)
Python has built-in most useful functions that does not need use regular expression. E.g. endswith() 

The generators can be implemented in C language. We can use "static" keyword, so local variables inside functions can retain the previous values as state of iterator. 

OOP language like Java, suggest to make attributes as private and then add getters and setters methods for them. The IDEs have support to write such methods automatically. In Python, by default the attributes are public. If needed, they can be converted as private property, and it does not impact the existing code. 

"Pythonic" is a new idiom. Let's see example of Pythonic API. Python has built-in urlib2 library. However, developing HTTP based client using urlib2 is less readable comparing developing the same using "requests" module. "requests" module is like "HTTP for humans". People talks a lot bout UI and UX. Python also focus on DX. DX means Developers' eXperience.  Have a look to these workshops about Pythonic APIs x.co/pythonic

The creator of Java programming language, wanted "inheritance" should be out of Java language. Julia is programming language for data science. Julia and Go, both programming languages do not support inheritance. 

Java and Python both have object member 'self' for all the member functions as an argument. 

The "language reference" document can be first place to understand any programming language. However, one may find "Python language reference" document as dry one. 

Key take away point: If you have not used "itertools" module, then most likely you might have written code, that was unnecessary. So study features of itertools Python module. 

Reference

Twitter : @ramalhoorg
E-mail : luciano.ramalho@thoughtworoks.com