Advertisements

• Categories

## Rough IT notes of no use for others

title: “Focused Learning compilation”
author: “Gopal Kumar”
date: “18 August 2015”
output: html_document

<!— YAML comments —>

# Learning MDown, pandoc, knitr, xeletex, R, postgreSQL, QGIS, Python, Andoridapps, CynagenMod
1. EMACS24, Markdown, pandoc, git, Hg, XeLeTeX, beamer, Impress.js,
2. R, knitr, statistics, Machine learning, Bigdata, Hadoop, SQL/Hive
3. SQL, postgreSQL, remotesensing, QGIS, RGIS
4. Python, Cpp, Java, JavaScript, Joomla, LAMPS[1],
5. Android Apps, CynagenMod, GSM4g
6. Linux, networking, KaliLinux, TCP/IP, linuxfromscretch, Cisco router program, switch, dns bind8,
7. IndustrialIP, Openbus Modbus, RespberryPi, Ardeno, Automation, Artificial intelligence
8. IndusrialDrives, Rotating Machines, sensors and actuators,, linear Motors, IGBT, IGCT,
9. Misc, HAMcode, make, scriptprogram
10. LibreOffice, Apache Server, ERP, webERP, webcollab
11. AutoCAD, FEA analysis, simulations,
12. Nature, advanture, travel, sustainable devlopment, law, history , geography, space, physics,

## Good sites with my login http://vk.com/datascience
## Basic UNIX commands http://www.datasciencecentral.com/group/resources/forum/topics/data-science-cheat-sheet
## install cygwin and then
You don’t need to spend hours learning UNIX Cygwin console:
cd, pwd, ls , tail -100, head -150 , cp, mv, mkdir, rmdir, wc, cat,
grep:
sort, uniq (sort alphabetically or numerically option)
gzip: compress/uncompress files
wc:
grep:
chmod:
history:
cron, crontab: schedule tasks (running an executable once a day)

> , >> to append , | (the pipe ), & (see section 2, used for background or batch mode ), * (see examples) and ! (see example)

## Book read and has some data http://www.win-vector.com/blog/introduction-to-data-science/
##github ggmtechn63

## Some sites
– datacentral <http://www.datasciencecentral.com/profiles/blogs/20-data-science-r-python-excel-and-machine-learning-cheat-sheets&gt;
– Large data sets for use <https://www.quandl.com/search&gt;
– Historical data visulaisation <http://101.datascience.community/&gt;
– some massive data mining softwares with more links site <http://www.kdnuggets.com/&gt;
Data Science
Data Science Cheat Sheet – Basic
Data Science Cheat Sheet – Advanced
Hadoop
Hadoop for Dummies cheatsheet
Getting Started Apache Hadoop Reference Card
Hadoop Command Line cheatsheet
Working with HDFS from the command line – Hadoop Cheat sheet
R
R cheat sheet (Google Drive)
R functions for Regression Analysis
R functions for Time series Analysis
R Cheat Sheet
Data Visualization with R
Data Analysis the data.table way
Data Visualisation with ggplot2 cheatsheet by R studio
Python
Python 2.7 Quick Reference Sheet
Python Cheat Sheet by DaveChild
Python Basics Reference sheet
NumPy / SciPy / Pandas Cheat Sheet

Machine Learning

Choosing the right estimator Machine Learning cheatsheet
Patterns for Predictive learning cheatsheet
Machine learning algorithm cheat sheet for Microsoft Azure
Machine Learning cheatsheet Github 1
Machine Learning cheatsheet Github 2
Machine Learning which algorithm performs best?
Cheat sheet 10 machine learning algorithms R commands

Things a Linux user must learn
Learn bash: Just read the complete man page of bash (man bash).
Learn vim: You might be using Emacs or Eclipse for your work all the time but nothing can compete vim.
Learn ssh: Learn the basics of passwordless authentication.
Learn basics of bash job management: Using &, Ctrl-C, fg, bg, Ctrl-Z, jobs, kill.
Learn basic commands for file management: ls and ls -l, less, head, tail and tail -f, ln and ln -s (hard links and soft links), chown, mount, chmod, df, du (du -sk *).
Learn basic commands for network management: dig, ifconfig.
Learn how to use grep, find and sed.
Learn how to use aptitude or yum (depends on the distro) to find and install packages.

For daily use
In bash, you may use Ctrl+R to search in command history.
In bash, you may use Ctrl+W to delete the last word, and Ctrl+U to delete the complete line.
Use cd – command to go back to the previous working directory.
Learn how to use xargs.

$find . -name \*.py | xargs grep some_function$ cat hosts | xargs -I{} ssh root@{} hostnameX

Use pstree -p command to get see the process tree. Learn various signals. eg to suspend a process, use kill -STOP [pid]. Type man 7 signal in terminal for complete guide.
If you want to keep running a background process forever then you can use nohup or disown.
Use netstat -lntp command to see what the processes are listening.
You should check about lsof also.
In your bash script you can use subshells to group commands.

# Do something in current dir

(cd /some/other/dir; other-command)

# Continue in original dir

Trimming of strings: ${var%suffix} and${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints “foo.txt”. The output of a command can be treated like a file via <(some command). For example, compare local /etc/hosts with a remote one: diff /etc/hosts <(ssh somehost cat /etc/hosts) Know about “here documents” in bash. Learn how to redirect both standard output and standard error via: some-command >logfile 2>&1. You should know about ASCII table (with hex and decimal values). Type man ascii in terminal. While working remotely via ssh, you should use screen or dtach to save your session. For web deveopers use of curl and curl -I, wget etc is useful. To convert HTML page to text file: lynx -dump -stdin If you must handle XML, xmlstarlet is good. In ssh, learn how to port tunnel with -L or -D (and occasionally -R). Also learn how to access web sites from a remote server. If you were typing a command but then changed your mind, Press Alt+shift+3. It will add # at the beginning and enter it as a comment. Data processing Learn about sort and uniq. Learn about cut, paste, and join. Learn how to get union, intersection and difference of text files. cat a b | sort | uniq > c # c is a union b cat a b | sort | uniq -d > c # c is a intersect b cat a b b | sort | uniq -u > c # c is set difference a – b Summing all numbers in the second column of a text file, code given below is probably 3X faster and 3X shorter than equivalent Python. awk ‘{ x +=$2 } END { print x }’

Learn about strings and grep command.
To split files into different parts learn about split (to split by size) and csplit (to split by a pattern).

System debugging

To know the status of your disk, cpu or network use iostat, netstat, top (or the better htop), and (especially) dstat.
To know your system’s memory status use free and vmstat command.
Use mtr which is a network diagnostic tool.
To find out which process or socket is using bandwidth, try iftop or nethogs.
You may use ab tool which is helpful for quick checking of web server performance.
For more serious network debugging take use of wireshark or tshark.
Learn how to use strace, and that you can strace a running process (with -p). This is helpful if your program is failing, hanging, or crashing, and you don’t know why.
Use the ldd command to check shared libraries.
Learn how to connect to a running process with gdb and get its stack traces.
Knowledge of /proc is very helpful. Examples: /proc/cpuinfo, /proc/xxx/smaps, /proc/xxx/exe, /proc/xxx/cwd, /proc/xxx/fd/.
When debugging why something went wrong in the past? To know about this use the sar command. It collects, reports and saves system activity information.
## EMACS24

OpenDir CX d, newfile CX-CF, save S, save as CW, Quit C, new window below CX 2, sidebyside CX 3, insertfile CX-i
screen CV MV
C bf np M bf np
select Cspc cut paste CY
#Short summary of important commands
***This is short summary of markdown commands***
————
*******
<gopalkumar@email.com>
[this is web address](www.yahoo.com)
# Rmarkdown shortnotes

—– or ********
~~strikeout~~
**bold** and *italics*
– itemised list
1. enumerate
<email>
[weblinktext]{weblinkaddress.com}
# H1
## H2
###### H6

Alternatively,
Alt-H1
======

Alt-H2
——

[I’m an inline-style link with title](https://www.google.com “Google’s Homepage”)

[I’m a reference-style link][Arbitrary case-insensitive reference text]

[I’m a relative reference to a repository file](../blob/master/LICENSE)

[You can use numbers for reference-style link definitions][1]

Or leave it empty and use the [link text itself]

ome text to show that the reference links can follow later.

[arbitrary case-insensitive reference text]: https://www.mozilla.org
[1]: http://slashdot.org
[link text itself]: http://www.reddit.com
Here’s our logo (hover to see the title text):

Inline-style:
![alt text](https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png “Logo Title Text 1”)

Reference-style:
![alt text][logo]

[logo]: https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png “Logo Title Text 2”
Markdowns highlighting many languages (and not-really-languages, like diffs and HTTP headers); see the highlight.js demo page.

Inline code has back-ticks around it.

Inline code has back-ticks around it.

Blocks of code enclosed by three back-ticks “
“javascript
var s = “JavaScript syntax highlighting”;
alert(s);
“

“python # If not indicated , no syntex highlight but can use html tags <b> tag </b>
s = “Python syntax highlighting”
print s
“

“
No language indicated, so no syntax highlighting.
But let’s throw in a <b>tag</b>.
“

var s = “JavaScript syntax highlighting”;
alert(s);

s = “Python syntax highlighting”
print s
**Tables
The outer pipes (|) are optional, and you don’no need to align raw Markdown line up. You can also use inline Markdown.

Markdown | Less | Pretty
— | — | —
*Still* | renders | **nicely**
1 | 2 | 3

**block quotes
> Blockquotes are very handy in email to emulate reply text.
> This line is part of the same quote. cascade also

Quote break.

**You can also use raw HTML in your Markdown, and it’ll mostly work pretty well.

Horizontal Rule

Three or more…

Hyphens

***

Asterisks

___

Underscores

Three or more…

Insert imagelinks in pure Markdown, but losing the image sizing and border:

<!— sample to be linked
[![IMAGE ALT TEXT HERE](http://youtube.com/YOUTUBE_VIDEO_ID_HERE/0.jpg)%5D(http://www.youtube.com/watch?v=YOUTUBE_VIDEO_ID_HERE)

—>

Lists Unordered You may use any of the following symbols to denote bullets for each list item:

* valid bullet
– valid bullet nested inside
+ valid bullet
Inline code

Wrap inline snippets of code with .

For example, <section></section> should be wrapped as “inline”.

For example, <section></section> should be wrapped as “inline”.

Indented code

Or indent several lines of code by at least four spaces, as in:
// Some comments
line 1 of code
line 2 of code
line 3 of code

Block code “fences”

Use “fences” “ to block in multiple lines of code.

“ html
Sample text here…
“

Sample text here…

HTML:

<pre>
<p>Sample text here…</p>
</pre>

Syntax highlighting

GFM, or “GitHub Flavored Markdown” also supports syntax highlighting. To activate it, simply add the file extension of the language you want to use directly after the first code “fence”, js, and syntax highlighting will automatically be applied in the rendered HTML. For example, to apply syntax highlighting to JavaScript code:

“ javascript
grunt.initConfig({
assemble: {
options: {
assets: ‘docs/assets’,
data: ‘src/data/*.{json,yml}’,
helpers: ‘src/custom-helpers.js’,
partials: [‘src/partials/**/*.{hbs,md}’]
},
pages: {
options: {
layout: ‘default.hbs’
},
files: {
‘./’: [‘src/templates/pages/index.hbs’]
}
}
}
};
“

Renders to this complicated HTML:
Tables

Tables by adding pipes as dividers and by adding a line of dashes, separated by bars beneath the header.
Pipe Need not vertically aligned.

| Option | Description |
| —— | ———– |
| data | path to data files to supply the data that will be passed into templates. |
| engine | engine to be used for processing templates. Handlebars is the default. |
| ext | extension to be used for dest files. |
Align: colon on one side of dashes below any heading will align text for that column.

| Option | Description |
| ——:| ———–:|
| data | path to data files to supply the data that will be passed into templates. |
| engine | engine to be used for processing templates. Handlebars is the default. |
| ext | extension to be used for dest files. |

Option Description
data path to data files to supply the data that will be passed into templates.
engine engine to be used for processing templates. Handlebars is the default.
ext extension to be used for dest files.
Links with title

[Upstage](https://github.com/upstage/ “Visit Upstage!”)

Renders to (hover over the link, there should be a tooltip):
Upstage HTML: <a href=”https://github.com/upstage/&#8221; title=”Visit Upstage!”>Upstage</a>

Named Anchors: enable you to jump to the specified anchor point on the same page. eg

# Table of Contents
* [Chapter 1](#chapter-1)
* [Chapter 2](#chapter-2)
* [Chapter 3](#chapter-3)

will jump to these sections:

## Chapter 1 <a id=”chapter-1″></a>
Content for chapter one.

## Chapter 2 <a id=”chapter-2″></a>
Content for chapter one.

## Chapter 3 <a id=”chapter-3″></a>
Content for chapter one.

NOTE that specific placement of the anchor tag seems to be arbitrary.
They are placed inline here since it seems to be unobtrusive, and it works.

##Footnotes
Type marker [^1]
Type the footnote key at the end of a long document.
[^1]: Cupcake Ipsum is fun text.

[^2]: [Cupcake Ipsum](http://www.cupcakeipsum.com/#)
##Images

Images have a similar syntax to links but include a preceding exclamation point.

![Minion](http://octodex.github.com/images/minion.png)

or

![Alt text](http://octodex.github.com/images/stormtroopocat.jpg “The Stormtroopocat”)

Like links, Images also have a footnote style syntax

![Alt text][id]

With a reference later in the document defining the URL location:

[id]: http://octodex.github.com/images/dojocat.jpg “The Dojocat”

The above cheatsheet noted from http://assemble.io/docs/Cheatsheet-Markdown.html
(the site about static blog genration?)
**Markup http://johnmacfarlane.net/pandoc/README.html**
**Write and publish a book**
Detailed writeup is very good at http://www.aristeia.com/authorAdvice.html
****
** Resource for Android**
Good resource and directions at http://wiki.cyanogenmod.org/w/Doc:_Development_Resources
**Ditch the MS word**
http://inundata.org/2012/12/04/how-to-ditch-word/
Softwares
Pandoc – format convertor
Mendeley refernce manager, export to bib
Markdown editor
Knitr to insert data tables

citations
cite this reference, add it in like so:
some statement [@Costello2009].
statement with multiple citations [@Costello2009; @Costello2010].
Compile With citations:
pandoc document.md -o document.pdf –bibliography citations.bib
With Formatting specific for a journal?
Download the citation styles from here and drop it into your folder. Then specify that style during document generation:
pandoc document.md -o document.pdf –bibliography cite.bib –csl style.csl
Do lot more like adding in results, tables, figures, and equations using mathjax but I’ll save the more advanced stuff for a future post.
———–
Usage: make [options] [target] …
Options:
–always-make Unconditionally make all targets.
–directory=DIRECTORY
–file=FILE, –makefile=FILE
–include-dir=DIRECTORY Search DIRECTORY for included makefiles.
–keep-going Keep going when some targets can’t be made.
–print-data-base Print make’s internal database.
———————–
****
Very low-level like the kernel, libc (aka bionic), and many Linux-ish parts in C.
Low leverl and 3rd-party in C or C++. ART (Andr Runtime for end-user programs), net tools, sound, shell, graphics drivers, etc.
The interactng user-facing Android “framework” like UI elements, most apps, in Java.

.mk files, Makefiles, and the /build directory, create a flashable .zip from source, primarily located in /build directory.
The various components/programs which together make up Android are each built independently through Android-specific Android.mk.
The Android.mk generally exists for each sub-project (or “module”) in its source-code directory.
This file directs the build system on exactly how to build that module, and where to put it in Android’s directory structure.
The files, once built, goes in /out/target/project/CODENAME directory (CODENAME is code name of device).
From there, they are zipped up and the final flashable (by recovery) .zip and flashable (by fastboot) .img files are produced.

You peek at what’s been built there in /out, as the directories that are turned into the .img and .zip files are still around.
In addition to the /build directory, the Android source code is organized into a hierarchy of folders.
Take a look here at a brief description of what’s where in the source code.
The $OUT directory Helpful Tip: After you build, you can type cd$OUT to automatically go to the /out/target/project/CODENAME directory.

kernel This is the kernel, obviously.
/system — all the stuff that will become the /system folder on Android.
/root — files that are turned into the ram disk loaded and run by the kernel. The first program to be run by the kernel is called init, and it uses the init.rc and init.CODENAME.rc files to determine what happens next. See an discussion of that here.

/recovery/root The ramdisk that contains the recovery mode is here.

Shortcut commands every CM dev should know(your computer, not device).
$. build/envsetup.sh — Note the “.” at the beginning. This load environment variables to your shell and aliases for shortcuts. know more about “$ . build/envsetup.sh” to know more about the breakfast, brunch and lunch commands,see Envsetup_help page

croot — this command will take you to the root of the source code.
mm and mm -B — this is the “make module” command
very useful if working on a particular module and don’t need to rebuild everything. cd into the directory that you want to test, then just type mm to build just the module in the working directory. to buid from scratch, add the -B.
This is a good companion with adb sync system below, which you can use to push the newly built files directly to your device for testing without having to reflash everything.

Make
make modules — this command will show all available targets. You can build a single one by make my_target.
make showcommands — this command will enable the verbose build mode.
ADB See also: adb
adb shell — command line shell inside your device.
adb reboot — quickly make your device reboot.
adb remount — If errored pushing files to /system due to it being in read-only mode, adb remount will remount /system into read-write mode— have root permissions. in liue of -o rw,remount /system (as root) or something.
diamonds$size[diamonds$carat >= 1] <- “Large”

## graphs
barplot(table(diamonds\$size), main=”Diamond Size Distribution”, xlab=”Size Category”, ylab=”Number of Diamonds”, col=”blue”)

Line charts
ggplot(diamonds, aes(clarity)) + geom_freqpoly(aes(group = color, colour = color)) +
labs(x=”Clarity”, y=”Number of Diamonds”, title=”Clarity by Color”)
Scatter plot:
ggplot(diamonds, aes(carat, price, color=clarity)) + geom_point() +
labs(x=”Carat Weight”, y=”Price”, title=”Price by Carat Weight”)
# scan reads vector
data <- matrix(scan(“birth.txt”), nrow=2, byrow=TRUE)

read.table(file=”clipboard”)

##
library(RODBC)
connection <- odbcConnect(“<DSN>”)
Once you have set up your connection, you could also use the sqlQuery() function to get data from .xls spreadsheets:
query <- “<SQL Query>”
data <- sqlQuery(connection, query)
str(data)
At the end of an R session, don’t forget to close the connections:
odbcCloseAll()

#DIF
Use the read.DIF() function to get your DIF files into R:

data <- read.DIF(“<your spreadsheet>”, header=FALSE, as.is = !stringsAsFactors)
# read xls
library(readxl)
read_excel(“<path to file”)

library(readODS)
read.ods(“<path to your file>”, sheet = 1, formulaAsFormula = FALSE)

library(jsonlite)
data <- fromJSON(“<Path to your JSON file>”)

For a well-explained quickstart with the jsonlite package, go

library(RJSONIO)
data <- fromJSON(“<Path to your JSON file”)
## For large data . This is different from the read.table(), which creates a data frame of your data.
library(data.table)
data <- fread(“http://assets.datacamp.com/blog_assets/chol.txt&#8221;)
library(data.table)
data <- fread(“http://assets.datacamp.com/blog_assets/chol.txt&#8221;, sep=auto, nrows = -1, na.strings = c(“NA”,”N/A”,””), stringsAsFactors=FALSE )

#sqldf
library(sqldf)
bigdata <- read.csv.sql(file=”<Path to your file>”,
sql=”select * from file where …”,
colClasses=c(“character”, rep(“numeric”,10)))

Certified Big Data Analyst Hadoop Certification Courses
This Big data analytics & hadoop training program extensively covers big data and predictive analytics techniques using R and Hadoop. Candidates will get practical hands-on training on cutting edge tools and big data platforms, like R and Hadoop (MapReduce, Hbase, Hive, Pig, Oozie, Scoope and Flume).
This big data online training is crafted by experts using real life business datasets. As part of this program candidates get access to the virtual lab and several case studies on big datasets for extensive hands-on practice. At end of the program candidate would need to operationalize and complete a live project for an assimilated learning.
Who should attend this Big data analytics hadoop courses & training program? MBA Students/ IT professionals/ Recent graduates who want job in big data analytics/ data scientist role.
Certified Big Data Analytics Course Content (72 hours + practice sessions)
Business Analytics using R & Tableau
Introduction to R- environment
1. The Workspace
2. Input/ Output
3. Useful Packages (Base & other packages) in R
4. Graphic User Interfaces (R studio)
5. Customizing Startup
6. Batch Processing
7. Reusing Results
Data Input & Output (Importing & Exporting)
1. Data Structure & Data Types (Vectors, Matrices, factors, Data frames,  and Lists)
2. Importing Data (Importing data from csv, txt, Excel and other files)
3. Keyboard Input (Creating input by entering data)
4. Database Input (Connecting to database and use the data)
5. Exporting Data (Exporting files into different formats)
6. Viewing Data (Viewing partial data and full data)
7. Variable & Value Labels –  Date Values
8. Missing Data
Data Management
1. Creating New Variables (calculations & Binning)
2. Operators (Using multiple operators)
3. Built-in Functions & User Defined Function
4. Control Structures(conditional statements, Loops)
5. Sorting Data
6. Merging and Appending Data
7. Aggregating Data
8. Reshaping Data
9. Sub setting Data
10. Data Type Conversions
Visualization
1. Creating Graphs
2. Histograms & Density Plot
3. Dot Plots –  Bar Plots – Line Charts – Pie Charts – Boxplots – Scatterplots
Basic Statistics (Exploratory Analysis)
1. Descriptive Statistics(central tendency/variance)
2. Frequency Tables /Summarization
3. Hypothesis Testing
4. t-tests/z-test (1-sample, independent sample, paired sample)
5. Analysis of Variance(ANOVA)
6. Correlations/chi-square test
Advanced Analytics (Advanced Statistics)
1. Introduction to predictive modeling & applications
2. Linear(Simple & Multiple) Regression
3. Logistic Regression
4. Introduction to segmentation
5. Segmentation using cluster analysis
Data Visualization using Tableau
1. Introduction to Tableau & Environment
2. Building basic views & sharing your work- overview
3. Data importing & manipulation
4. Maps/Tables/Calculated fields
5. Parameters
6. Data visualization with Charts maps
7. Building & customizing Reports
8. Building & customizing Dashboards
Machine Learning using R
1. What is Machine Learning?
2. Applications of Machine Learning Algorithms
3. Classification & Regression Problems
4. Training & Testing concepts – Cost & optimization functions
5. Artificial Neural Networks(ANN)
6. Support Vector Machines(SVM)
7. Decision Tress & Random Forest
8. Baysian Network case
Social Media Analytics using R
1. Social Media – Characteristics of Social Media
2. Applications of Social Media Analytics
3. Metrics(Measures Actions) in social media analytics
4. Examples & Actionable Insights using Social Media Analytics
5. Text Analytics – Sentiment Analysis using R
6. Text Analytics – Word cloud analysis using R
Projects (Applying Overall Learning)
1. Solve Business problems using R/Tableau
HADOOPIntroduction to Big Data & Hadoop
1. What is Big Data?
2. Types of Data
3. Characteristics of Big Data
4. Need for understanding Big Data (Application of Big Data)
5. Traditional Approaches and its limitations
6. Introduction to Hadoop and eco-system
7. Getting Started with Hadoop (software installation etc.)
Hadoop Architecture
1. Hadoop Commercial version vs Apache Hadoop
2. Hadoop Cluster in commodity hardware
3. Hadoop core components
4. HDFS layer
5. HDFS operation principle
6. Basic Hadoop commands
MapReduce
1. Introdution to MapReduce
2. Hadoop MapReduce example
3. Hadoop MapReduce Characteristics
4. Setting up your MapReduce Environment
5. Building a MapReduce Program
6. Input Formats in MapReduce
7. OutputFormats in MapReduce
8. Basic MapReduce Programming using R
R-Hadoop
1. Introduction to RHdfs, Rmr and Rhbase
2. Develop Map reduce code using R for Local & Hadoop env
3. Exploratory analysis using R-Hadoop
4. Predictive analytics using R- Hadoop
5. Overview of Parallelization using R without Hadoop
Introduction to Flume & Sqoop
1. Introduction to Sqoop (Why, what, processing, under the hood)
2. Exporting data from Hadoop using Sqoop
3. Introduction to Flume
4. Flume Use Cases
5. Hands on Exercise using Flume and Sqoop
PIG
1. Introduction to PIG
2. Components of PIG
3. PIG Data Model
4. Creating Mapreduce programs using PIG
5. Hands on Exercise using PIG
HIVE
1. Introduction to HIVE and its characteristics
2. Components of HIVE
3. HIVE Data Models
4. Serialization/De-serialization
5. HIVE file formats
6. HIVE Query Language
7. HIVE Functions
8. Difference between HIVE and PIG
9. Hands on Exercise using HIVE
H-Base
1. HBase introduction and its Characteristics
2. HBase Architecture
3. Storage Model of HBase
4. When to use HBase
5. HBase Data Model
6. HBase Families
7. HBase Components
8. Data Storage
9. Hands on Exercise using Hbase
Mahout
1. Mahout introduction and its Characteristics
2. Mahout Architecture
3. When to use Mahout
4. What are the Machine Learing topics are covered in Mahout
5.  Hands on Exercise using Mahout
ZooKeeper
1. Introduction to ZooKeeper & its Features
2. Features of ZooKeeper
3. Challenges faced in distributed applications
4. Coordination
5. ZooKeeper: Goals and Uses
6. ZooKeeper: Entities, Data Model, Services
Misc Components
1. Overview of Apache Oozie
2. Overview of Storm
3. Overview of Apache Cassandra
4. Overview of Apache Spark
5. Overview of H2O
6. Social Media Analytics(Text Analysis, Word cloud)
Projects (Applying Overall Learning)
1. Solve Business problems using all the components of Hadoop

AnalytixLabs now also offers online Big data analytics & hadoop courses training in Delhi, India.

Advertisements

## Links on export incentives procedures

http://help.sap.com/saphelp_47x200/helpdata/en/93/7441c2546011d1a7020000e829fd11/frameset.htm

## Import Procedure shot Indian customs

Procedure for Import and Export

ENTRY – ‘Entry’ : an entry made in a Bill of Entry, Shipping Bill or Bill of Export. It includes
(a) label or declaration accompanying the goods which contains description, quantity and value of the goods, in case of postal articles u/s 82
(b) Entry to be made in case of goods to be exported
(c) Entry in respect of goods imported which are not accompanied by label or declaration made as per provisions of section 84.[section 2(16)].

AMENDMENT TO DOCUMENTS -Importer, exporter or ‘Person In charge’ have to submit documents to customs authorities like Bill of Entry, Import Manifest, Export Manifest etc.
Document modification only with prior permission [section 149] for reasons like change in classification, clerical mistake, change in unloading/ loading plan of vessel.
In case of bill of entry, shipping bill or bill of export, amendment after clearance only on the basis of documentary evidence which was in existence when goods were cleared, warehoused or exported, and not on basis of any subsequent document.

Customs Station – defined : Imported goods unloading permitted only at specified places. Similarly, export permitted only from specified area.
Customs area means all area of Customs Station and includes any area where imported goods or export goods are ordinarily kept pending clearance by Customs authorities.
Customs Station means (a) customs port (b) inland container depot (c) customs airport and (d) land customs station.

Section 7 of Customs Act empowers CBEC (Board) to appoint * Customs ports * Customs airports * Places for ICD’s * Coastal ports by issuing a notification.
Section 8 authorises Commissioner of Customs to approve proper places for unloading and loading of goods and specify the limits of customs area.

Import Procedures

‘person-in-charge of conveyance’ as well as the importer procedures.(section 2(31),vessel master;air commander; train guard; vehicle I/c/driver.
The significant definition for responsiblety for -submitting Import Manifest and Export Manifest ; follow approved route and lands at approved place only.
-goods are unloaded/loaded after written order, at proper place. -conveyance does not leave without written order of Customs authorities.
He can be penalised for (a) Giving false declaration and statement (b) shortages or non-accounting of goods in conveyance.
The ‘person in charge of conveyance’ (carrier of goods) has to follow prescribed procedure.
-Arrival at customs port/airport only -U/Section 29 entering India shall call or land at customs port or customs airport only.
-Landing at other place only in emrgency of accident, stress of weather or other unavoidable cause and report to nearest police station or Customs Officer.
-While arriving by land route, the vehicle should come by approved route to ‘land customs station’ only.
Import Manifest/ Report/IGM – Person-in-charge of vessel, aircraft or vehicle has to submit Import Manifest / Report. [termed as IGM – Import General Manifest].
(In case of a vessel or aircraft -> called import manifest, For Vehicles, it is called import report.) Section 30(1)
– IMPORT MANIFEST TO BE SUBMITTED BEFORE ARRIVAL OF AIRCRAFT OR VESSEL. Import report (for vehicle) to be submitted within 12 hours of arrival at the customs station.
Penality for Delay:  person-in-charge  upto Rs 50,000. Condoned if the excise officer is satisfied about sufficiency of cause for the delay. [section 30(1)].
Normally, Agents submit before arrival of vassal/aircraft and complete maximum formalities. They thus also file ‘Bill of Entry’ in advance.

Grant of “Entry Inwards” by Customs Officer for Unloading of cargo. Date significant applicable customs duty. Granted only when berthing accommodation is granted
Carrier responsible for shortages during unloading –
If the goods are short landed, the carrier is liable to pay penalty upto twice the amount of duty payable on such short landed goods.
Tally sheet prepared by Port Trust authorities on unloading of goods is a statutory document and  accepted in preference to steamer survey.

Procedures for Importers -The importer importing the goods has to follow prescribed procedures for import by ship/air/road.
Bill of Entry -This is a very vital and important document which every importer has to submit under section 46 in prescribed form.
(The standard size is 16″ × 13″ and 15″ × 12″ for computerised BoE vide Mumbai Customs No. 142/93 dated 3-1193).
Bill of Entry in quadruplicate – original and duplicate for customs, triplicate for the importer and fourth copy is meant for bank for making remittances.
Under EDI system, Bill of Entry is actually printed on computer in triplicate only after ‘out of charge’ order is given. Duplicate copy is given to importer.
WHITE   BILL OF ENTRY FOR HOME CONSUMPTION  -cleared on payment of full duty.
YELLOW  BILL OF ENTRY FOR WAREHOUSING -(INTO BOND CLEARANCE) -store the goods in warehouse without payment of duty under a bond.
GREENBILL OF ENTRY FOR EX-BOND CLEARANCE -for clearance from warehouse on payment of duty. [goods already classified and value assessed but duty rate of day].
BIN on Bill of Entry – 15 digit (includes 10 digit PAN) BIN (Business Identification Number) allotted to importer and exporter w.e.f. 1.4.2001. (earlier IE Code)
Filing of Bill of Entry -Normally filed by CHA on behalf of the importer. Electronic filing at computerised port through Customs EDI.
Procedure vide Bill of Entry (Electronic Declaration) Regulations, 1995.

Documents to be submitted by Importer -Documents to enable customs to (a)check the goods (b)decide value and classification of goods and (c)import legally permitted.
The documents that are essentially required are :
(i) Invoice
(ii) Packing List
(iii) Bill of Lading / Delivery Order
(iv) GATT declaration form duly filled in
(v) Importers / CHAs declaration duly signed
(vi) Import Licence or attested photocopy when clearance is under licence
(vii) Letter of Credit / Bank Draft wherever necessary
(vii) Insurance memo or insurance policy
(viii) Industrial License if required
(ix) Certificate of country of origin, if preferential rate is claimed.
(x) Technical literature.
(xi) Test report in case of chemicals
(xii) Advance License / DEPB in original, where applicable
(xiii) Split up of value of spares, components and machinery
(xiv) No commission declaration.
– A declaration in prescribed form about correctness of information.– Chapter 3 Para 6 and 7 of CBE&C’s Customs Manual,2001.

The Noting is now done electronically in large ports,  manually noting in small ports. Thoka Number (Serial Number) is given while noting the Bill of Entry.

Electronic submission under EDI system –
Formal submission of Bill of Entry is not required, as it is generated in computer system.
Importer should submit declaration in electronic format to ‘Service Centre’.
A signed paper copy of declaration for non-repudiability should be submitted.
Bill of Entry number is generated by system which is endorsed on printed check list.
Original documents are to be submitted only at the stage of examination.
Assessment of Duty and Clearance
The documents submitted by importer are checked and assessed by Customs authorities and then goods are cleared.
Section 2(2) defines ‘assessment’ as follows – ‘Assessment’ includes provisional assessment, reassessment and any order of assessment in which the duty assessed is Nil.

Noting of Bill of Entry -Bill of Entry submitted by importer or CHA is cross-checked with ‘Import Manifest’ submitted by person in charge of vessel / carrier.
‘Noting’ really means taking on record by customs officer. Relevant for determining rate of customs duty.
Thoka number (serial number) is given in the import section. Otherwise, it is returned for clarifications. In EDI system, noting and entry number done by system itself.

Date of presentation of bill of entry is highly relevant for rate of duty.

Bill of Entry is accepted only after proper scrutiny vis-a-vis import manifest and various declarations given in bill of entry and attached documents like invoice, bill of lading etc. Else the authorities can refuse to accept the Bill of Entry, and hence submission of such incomplete Bill of Entry cannot be taken as date of presentation of Bill of Entry – Simla Agencies v. CC 1993 (63) ELT 248 (CEGAT).

Prior Entry of Bill of Entry -After the goods are unloaded, these have to be cleared within stipulated time usually three working days.
If these are not so removed, demurrage is charged by port trust/airport authorities, which is very high.
Hence, importer complete as many formalities as possible before ship arrives. Section 46(3) of Customs Act allows importer to present bill of entry upto 30 days before expected date of arrival of vessel. In such case, duty will be payable at the rate applicable on the date on which ‘Entry Inward’ is granted to vessel and not the date of presentation of Bill of Entry, but rate of exchange will be as prevalent on date of submission of bill of entry. – confirmed in CC, New Delhi circular No 64/96 dated 10.12.1996 and CBE&C circular No 22/97-Cus dated 4.7.1997.

Assessment of Customs duty -Section 17 provides that assessment of goods will be made after Bill of Entry is filed.
Date stamp of receipt is put on the ‘Bill of Entry’ and then it is sent to appraising department either manually or electronically

There are various Appraising groups for different Chapter headings. Each group is under an Assistant/Deputy Commissioner. Group consists of ‘Examiners’ and ‘Appraisers’.

APPRAISING THE GOODS -Appraiser has to
(a) correctly classify the goods
(b) decide the Value for purpose of Customs duty
(c) find out rate of duty applicable as per any exemption notification and
(d) verify that goods are not imported in violation of any law.
He can call for any further documents that may be required for assessment.
and if required, issue an examination order, usually on the reverse of Bill of Entry. If such order is issued,
the Bill of Entry is presented to appraising staff at docks / air cargo complexes, where the goods are examined in presence of importer’s representative.
Assessment is finalised after getting the report of examination. – Chapter 3 Para 11 and 12 of CBE&C’s Customs Manual, 2001.

VALUATION OF GOODS – As per rule 10 of Customs Valuation Rules, the importer has to file declaration about full ‘value’ of goods.
If the assessing officer has doubts about the truth and accuracy of ‘value’ as declared, he can ask importer to submit further information, details and documents.
If the doubt persists, the assessing officer can reject the value declared by importer. [rule 10A(1) of Customs Valuation Rules].
If the importer requests, the assessing officer has to give reasons for doubting the value declared by importer. [rule 10A(2)].
If the value declared by importer is rejected, the assessing officer can value imported goods on other basis e.g. value of identical / similar goods etc.
[amendment w.e.f. 19.2.98, as per WTO MF(DR) circular No. 16/2003-Cus dated 17-3-2003. Burden of proof on department to not arbitrarily reject/ increase value.

APPROVAL OF ASSESSMENT – For value over Rs One lakh to be approved by Asst.Commissioner. ( Cases covered under ‘fast track clearance -appraiser authorised)
After the approval, duty payable is typed by a “pin-point typewriter” so that it cannot be tampered with. As per CBE&C circular No. 10/98-Cus dated 11-2-1998, Assessing Officer should sign in full in Bill of Entry followed by his name, preferably by rubber stamp.

EDI ASSESSMENT – In the EDI system, the cargo declaration is transferred to assessing officer in the groups electronically. Screen Processing. Calculations by system.
If assessing officer needs clarification, he can raise a query. The query is printed at service centre and importer replies through service centre.
Facility of tele-enquiry about status of documents is provided in major customs stations.
Under EDI, normally, documents are inspected only after assessment. After assessment, copy of Bill of Entry is printed at service centre.
Final Bill of Entry is printed only after ‘Out of Charge’ order is given by customs officer. – Chapter 3 Para 18 to 22 of CBE&C’s Customs Manual, 2001.

PAYMENT OF CUSTOMS DUTY -After assessment of duty, duty is paid. Regular importers/CHA keep PDA. Duty debited to such PDA account, or in cash/DD through TR-6 challan.

After payment of duty, if goods were already examined, delivery of goods can be taken from custodians (port trust) after paying their dues.
If goods were not examined before assessment, these have to be submitted for examination in import shed to the examining staff.
After shed appraiser gives ‘out of charge’ order, delivery of goods can be taken from custodian.

First and second system of assessment -There are two systems of assessment.
Section 17(2) provides for assessment after examination of goods
Section 17(4) provides for assessment on basis of documents, followed by inspection and testing of goods.

“First appraisement system” or ‘first check procedure’ is followed if the appraiser is not able to make assessment on the basis of documents submitted and deems that inspection is necessary. Goods are examined first and then these are assessed. This method is followed only if assessment is not possible on basis of documents. – -The importer himself may also request ‘first check procedure’, if he cannot give all required details regarding description / value of goods. He has to make request for first check examination at the time of filing of Bill of Entry or at data entry stage in case of EDI. He has to give reason for seeking first appraisement. The examination order is recorded on Bill of Entry and then returned to importer / CHA. It is then presented to import shed for examination. The shed appraiser / Dock examiner examines the goods as per examination order and records his findings. If samples are required, they are taken out. In case of EDI
system, the report of examination is given in the computer itself. The goods are then assessed to duty by appraiser. – Chapter 3 Para 23 of CBE&C’s Customs Manual, 2001.

In “Second Appraisement System” or ‘second check procedure’, which is normally followed, assessment is done on basis of documents and then goods are examined.
Such examination is not mandatory. It is done on selective basis on the basis of ‘risk assessment’ or specific intelligence report. Section 17(4) of Customs Act specifically provides that if initially assessment is done on basis of documents, re-assessment can be done after examination or testing of goods or otherwise, if it is found subsequent to examination or testing or otherwise, that any statement made on Bill of Entry or any information supplied is not true in respect of matter relevant to assessment of duty.

First appraisement is generally carried out in following cases – * If complete documents are not submitted *
Goods are to be tested for correct classification * Goods are re-imported * Goods are damaged or deteriorated and abatement is claimed * Goods are abandoned and remission of duty is applied for * When goods are provisionally assessed * When importer himself requests for examination of goods before payment of duty.

EXAMINATION OF GOODS -Examiners carry out physical examination and quantitative checking like weighing, measuring etc. Selected packages are opened and examined on sample basis in ‘Customs Examination Yard’. Examination report is prepared by the examiner.

Accelerated Clearance of Imports and Exports Scheme (ACS) – MF(DR) circular No. 30/2003-Cus dated 4-4-2003 on trial at Sahar (Mumbai), ICD/TKD and Chennai Port.
Importer will himself determine classification of goods including claim for exemption benefits. Computer System will calculate the duty based on his declaration.
Physical inspection done by risk-assessment and management techniques on a computer based system and not on the orders of customs examining staff. Audit of import documents will not be by existing system of concurrent audit but will be done by post-clearance audit, as prevalent in developed countries.

In case of imports, the scheme will be open to all status holders under EXIM policy, Central and State Government PSUs and other importers who have been importing for at least two years and have filed at least 25 Bills of Entry in preceding year. – –

In case of exports, the scheme open to all status holders under EXIM policy, EOU/STP/EHTP units whose goods have been sealed in presence of customs/excise officers,
Central and State Government PSUs, manufacturer-exporters who have been exporting for at least two years and have filed at least 25 Shipping Bills in preceding year and bulk exporters. – – Certain sensitive items have been excluded from the provisions.
Importer/exporter intending to avail this facility has to make application to Commissioner. The clearances will be subject to post clearance audit.

Provisional Assessment -Section 18 of Customs Act, 1962 provide that provisional assessment can be done in following cases
(a) when Customs Officer is satisfied that importer or exporter is unable to produce document or furnish information required for assessment
(b) it is deemed necessary to carry out chemical or other tests of goods
(c) when importer/exporter has produced all documents, but Customs Officer still deems it necessary to make further enquiry.

The importer/exporter has to furnish guarantee/security as required by Customs Officer for payment of difference if any.
Goods can be cleared after payment of duty provisionally assessed and after providing the security.
After final assessment, difference is paid by importer or refunded to him as the case may be.
If the imported goods were warehoused after provisional assessment, the Customs Officer may require importer to execute a bond for twice the
difference in duty, if duty finally assessed is higher [section 18(2)(a)]. The bond is called as ‘P D Bond’ (Provisional Duty Bond).
The bond is with security or surety. Bank guarantee can also be given as a security.

Checking of duty drawback / license documents -Documents in respect of Duty Entitlement Pass Book (DEPB), advance license, duty drawback etc. will be checked.

Execution of bond and payment of duty -Once the duty is assessed, the bill of entry is returned to importer.
The Bill of Entry should be presented to comptist for calculation and pinpointing of the duty. If bond has to be executed, it will be taken in bond section.

Payment of duty – If goods are to be removed to a warehouse under bond, duty payment is not required. Payment required If goods are to be removed for home consumption.
Large importers and CHA have P.D. accounts with customs. (provisional duty account. is a current account, similar to PLA in central excise.)
The importer or CHA pays lump sum amount in the account and can pay customs duty by debiting the amount in P.D. (Provisional Duty) account.
If no PDA account, he can pay duty by cash using TR-6 challan.
The duty to be paid within five working days after the ‘Bill of Entry’ is returned to the importer otherwise interest payable. [section 47(2)].
Such interest can be between 10% to 36% as may be notified by Central Government. [Section 47(2) of Customs Act, 1962.]. Interest rate is 15% w.e.f. 13-5-2002.

Disposal if goods are not cleared within 30 days after unloading -section 48 of Customs Act. Customs Officer can grant extension.
Else goods can be sold after giving notice to importer.
However, animals, perishable goods and hazardous goods can be sold any time – even before 30 days. Arms & ammunition sold only with permission of Central Government.

Out of Customs Charge Order –
After goods are examined, it is verified that import is not prohibited and after customs duty is paid, Customs Officer will issue ‘Out of Customs Charge’ order under section 47. Goods can be cleared from customs area only on receipt of such order. An ‘adjudicating order’ of Customs Act, even if it is passed by Appraiser.

Demurrage if goods not cleared -Heavy demurrage is payable if goods are not cleared from port within three days.

Import of software through data communication – ??? MF(DR) circular No. 58/2000-Cus dated 10-7-2000.

Relevant Date for Rate and Valuation of Customs Duty – Section 15
duty and tariff valuation shall be one of the following dates.
(a) if the goods are entered for home consumption, the date on which bill of entry is presented
(b) in case of warehoused goods, when Bill of Entry for home consumption is presented u/s 68 for clearance from warehouse and
(c) in other cases, date of payment of duty.

## Cholesterol and carbohydrates bad

Information noted from http://cholesterol.about.com
carbohydrates and cholesterol are not just trouble individually, but they also work together in harmful ways., researchers have prooved high-carb diet raise levels of cholesterol in the blood.
What Exactly is Cholesterol?
Cholesterol is a waxy fat that is essential to cell membranes and brain function. But if the body has too much “bad cholesterol,” known as LDL, the excess is deposited into the arteries. On the other hand, “good cholesterol”, or HDL, works like a clean-up crew in the bloodstream, ferrying LDL to the liver for safe disposal.

Total cholesterol tests measures blood levels of HDL, LDL, and triglycerides (the main form of fat in food and in the body). High levels of LDL and triglycerides increase the risk for heart disease and stroke, while a high level of HDL cholesterol is considered heart-healthy.

So Where Do Carbohydrates Fit In?

Bad carbs are simple carbohydrates, such as refined sugars, white flour, potatoes, and white rice. Such foods are digested quickly and trigger a burst of insulin, the main hormone responsible for converting food into energy. But the more insulin that is produced, the more likely that excess calories will be stored as fat, resulting in higher triglyceride and LDL levels.

Good carbs are complex carbohydrates, such as whole grains, brown rice, dried beans, fruits, vegetables, and nuts. These foods are digested more slowly, so they do not cause an insulin jolt and are less likely to boost LDL and triglyceride levels.

Unfortunately, bad carbohydrates make up a large part of the typical American diet. These simple carbohydrates are found in white bread, baked and mashed potatoes, french fries, sugar-sweetened beverages, and most cookies, pizzas, and pastas.

diets high in bad carbohydrates increase a woman’s risk for developing heart disease. the impact of bad carbs on cholesterol emphasizes the need to go beyond limiting fat intake as a strategy for heart health.

recommends several steps:

Choose whole grain foods over refined grains.
Avoid sugar-sweetened beverages, which are the source of almost half the added sugar in the diet.
Replace carbohydrates (especially sugars and processed flour) with healthy proteins such as beans, peas and nuts, as well as unsaturated fats, such as olive oil, canola oil and other vegetable oils.

## World Bank , FIDIC, NEC and ENAA contracts links

Some source for World Bank, FIDIC, NEC and ENAA contracts links

Contract documents

FIDIC

NEC

Details fo ENNA forms from their website http://www.enaa.or.jp/EN/activities/model.html

 ENAA Model Form-International Contract for Power Plant Construction(Turnkey Lump-sum Basis) was also published in 1996 to meet the growing needs for an international contract model form for construction of power plants.In preparing the Model Form, the Committee extensively referred to, and took into consideration, the comments, recommendations, advice and suggestions of various sources such as the World Bank, and other major financing institutions, potential customers and contractors, other relevant organizations in the US and Europe. The Model Form also received a thorough check by UK law firms and a certified quantity surveyor. This 2010 edition does not include any “Appendix” or “Work Procedures” such as were provided in the 1992 edition. Only Appendix 8 “Form of Bonds” has been revised this time and contained in this edition. All other Appendices and Work Procedures of the 1992 edition are intended to remain in use with this edition unchanged. Compiled with flexibility and fair and reasonable balance between the Owner and the Contractor on the various risks involved in international projects, ENAA Model Form is intended for a wide range of users, including in-house legal and sales personnel and those who are involved in the various phases of actual project implementation.

Process Plant Model Form
The Model Form for Process Plant Construction is the core of ENAA’s series of model forms and since its first edition, it has been highly acclaimed by the industry and used extensively for international projects related to construction of process plants. With some modifications, it has also been used as the General Conditions of Contract for the World Bank’s Standard Bidding Documents “Supply and Installation of Plant and Equipment”.The revisions to the 1992 Form adopted in this edition are made in the light of many invaluable comments from a large audience. More than ten years have passed since we brought out the 1992 Form. During that period, the scale of projects has grown in size and the financing schemes for such large projects have become extremely complex, thereby increasing the number of stake-holders who seek more clarity in the allocation of risks and responsibilities. We felt it was our duty to respond to such needs of the industry and to produce a revised version to fulfill such needs.

ENAA considered many factors in preparing this edition but tried to maintain the risk allocation originally envisaged in the first edition thus limiting the possibility of any confusion among the users accustomed to using the previous editions.

Contents – Process Model Form – 2010 Edition:

 Form of Agreement and General Conditions Form of Agreement General Conditions Part I Contract and Interpretation Part II Subject Matter of Contract Part III Payment Part IV Intellectual Property Part V Work Execution Part VI Guarantees and Liabilities Part VII Risk Distribution Part VIII Change in Contract Elements Guide Notes Chapter I. The General Concept of the ENAA Model Form Chapter II. Guide Notes for Agreement Chapter III. Guide Notes for General Conditions Chapter IV. Application to the case where No Process License is required from Contractor

For reference (Vol.2, Vol.4 and Vol.5, 1992 Edition):

 Vol.2 Sample of Appendices to the Agreement Appendix 1 Breakdown of the Contract Price Appendix 2 Escalation Clause Appendix 3 Unit Rates for Changes Appendix 4 Payment Terms Appendix 5 Form of Letter of Credit/Guarantee Appendix 6 Insurance Requirements Appendix 7 Performance Guarantees Appendix 8 Form of Bonds Appendix 9 Project Specifications Vol.4 Work Procedures WP 1 Correspondence Procedure WP 2 Payment Application Procedure WP 3 Approval and Review Procedure WP 4 Work Change Procedure WP 5 Procurement Procedure WP 6 Expediting Procedure WP 7 Shop Inspection Procedure WP 8 Field Inspection Procedure WP 9 Progress Report Procedure WP 10 Commissioning and Performance Test Procedure Vol.5 Form of Agreement and General Conditions (Alternative Form for Industrial Plant – Without Process License) Form of Agreement General Conditions Part I Contract and Interpretation Part II Subject Matters of Contract Part III Payment Part IV Intellectual Property Part V Work Execution Part VI Guarantees and Liabilities Part VII Risk Distribution Part VIII Changes in Contract Elements Schedule 1 Form of Mechanical Completion Certificate Schedule 2 Form of Acceptance Certificate

Power Plant Model Form In the course of preparation of the ENAA Process Form, it was recognized that there was a growing need for a model form of an international contract for power plant construction. It was also recognized that power plants are a fundamental part of the infrastructure of industrial society and also play a major role in the development of society in general, and of industry in particular, in developing countries.In drafting this Model, the Committee has endeavored to incorporate some ideas, which are fair, impartial and practical, but not generally contained in other model forms for power plant construction contract. It makes this Model Form, we believe, all the more useful for all related parties.

Main Features:
(1)   Most suitable for use in a full turnkey power plant construction projects.
(2)   Fair and reasonable balance between the Owner and the Contractor on the various risks involved in international projects on a full turnkey basis.
(3)   More focus is provided on and improvement made with regard to the provisions relating to Taking Over procedures such as Commissioning and Reliability Test and also the provisions relating to the Performance Guarantees.
(4)   Notes for construction contracts under project finance scheme are added to the Guide Note as an appendix.

Contents – Power Model Form-:

 Vol.1(112 pages) Form of Agreement and General Conditions Form of Agreement General Conditions Part I Contract and Interpretation Part II Subject Matters of Contract Part III Payment Part IV Intellectual Property Part V Work Execution Part VI Guarantees and Liabilities Part VII Risk Distribution Part VIII Changes in Contract Elements Vol.2(80 pages) Sample of Appendices to the Agreement Appendix 1 Breakdown of the Contract Price Appendix 2 Escalation Clause Appendix 3 Unit Rates for Changes Appendix 4 Payment Terms Appendix 5 Form of Letter of Credit/Guarantee Appendix 6 Insurance Requirements Appendix 7 Performance Guarantees Appendix 8 Form of Bonds Appendix 9 Project Specifications Vol.3(36 pages) Guide Notes Item 1. Features of the ENAA Model Form and its Fields of Application Item 2. Engineer Item 3. Completion/Taking Over and Performance Guarantee Item 4. ContractorÅfs Total Maximum Liability Item 5. Training Item 6. Sub-contracting Item 7. Environmental Conditions

The Model Forms are available either in document format (A4 booklet) or CD (in MS-Word 2000 version).
>>Process Model Form & Power Model Form ODER FORM

EPS Model Form Both Process Plant and Power Plant model forms are on a turnkey lump sum basis. A further type of contract, however, still used in many cases for international construction projects is the Engineering, Procurement and Supply (EPS) type contract, for which only a limited number of model forms exist. ENAA has noted the need for a model form for EPS type contract and to fulfill such need.In this new EPS model form, ENAA decided to refrain from making amendment to the general approach of the existing model forms other than to accommodate the changes necessarily flowing from the change of the contract scheme to the EPS scheme. ENAA has deferred the making of any such amendments until publication of the revised edition of the existing model forms that will follow this new EPS model form.

Contents – EPS Model Form-:

 Agreement General Conditions Part I Contract and Interpretation Part II Subject Matter of Contract Part III Payment Part IV Intellectual Property Part V Work Execution Part VI Guarantees and Liabilities Part VII Risk Distribution Part VIII Change in Contract Elements Schedule 1 Delivery Certificate Schedule 2 Acceptance Certificate Guide Notes Chapter I. General Chapter II. Guide Notes for Agreement Chapter III. Guide Notes for General Conditions

EPS Model Form includes CD as a part of document.
>>>EPS Model Form ORDER FORM

World Bank information from

http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:20062773~menuPK:84284~pagePK:84269~piPK:60001558~theSitePK:84266,00.html

Details at world bank site Home > Projects > Procurement > Index of Standard Bi… > Supply and Installation – Index

PrefaceThese “Standard Bidding Documents for the Supply and Installation of Plant and Equipment” have been prepared for use in contracts financed by the International Bank for Reconstruction and Development (IBRD) or the International Development Association (IDA) (interchangeably known as “The World Bank” or “the Bank”) involving the supply, installation and commissioning of specially engineered plant and equipment, such as turbines, generators, boilers, switchyards, pumping stations, telecommunications, process and treatment plants, and the like for power, water, sewerage, telecommunication and similar projects. Normally, this set of documents should be used when (i) the value of the plant and equipment portion represents the major part of the estimated contract value, or (ii) the nature and complexity of the plant and equipment is such that the facilities cannot safely be taken over by the Employer without elaborate testing, precommissioning, commissioning and acceptance procedures being followed. If the user has questions regarding which SBD should be used in a particular case, the appropriate Bank’s Regional Procurement Advisor or the Procurement Policy and Services Group, Operational Core Services Network, should be consulted.

These documents include both a Single Stage (Option A) and an alternative Two Stage (Option B) bidding procedure to be chosen by Borrowers as appropriate to each circumstance. The choice of the Two Stage bidding procedure shall be subject to the Bank’s advance approval on a case-to-case basis, depending on the complexity of the contract and the particular circumstances surrounding its procurement and implementation. The Bidding Procedures Flowcharts (one for Single Stage bidding and one for Two Stage bidding) show how the bidding documents are intended to be used both during the bidding process and during the formulation of the eventual contract documents.

(i) The use of SBDs prepared by the Bank is mandatory for all contracts financed by the Bank. (ii) The documents have been prepared as standard documents, which can be used in their published form without the need for the Borrower to amend or add text to the standard sections of the document. All information and data particular to each individual contract and required by bidders in order to prepare responsive bids must be provided by the Employer, prior to issuing the Bidding Documents, in the Bid Data Sheet (Section III), the Special Conditions of Contract (Section V), the Technical Specifications and Drawings (Section VI), and the Appendixes to the Form of Contract Agreement (in Section VII). Unless specifically agreed with the Bank, the Special Conditions of Contract shall not materially alter the provisions of the General Conditions of Contract. (iii) The role of “the Engineer,” as found, for example, in the Conditions of Contract for Electrical and Mechanical Works produced by the International Federation of Consulting Engineers (FIDIC), is not included in these Bidding Documents. In its place, and fulfilling the same duties, there are two new parties named the “Project Manager” and the “Adjudicator.”

The Project Manager is appointed by the Employer. Its role is to supervise and manage the Contract on behalf of the Employer with the intention of achieving the Employer’s objectives for the completed Contract. When appointing the Project Manager, the Employer may either select a reputable firm of Consulting Engineers experienced in the particular field, or the Employer may appoint one of its own staff as Project Manager, if the Bank is satisfied that it has suitable in-house expertise.

The name of the Adjudicator proposed by the Employer is to be given in the Bid Data Sheet, and the bidders have the right to accept or propose an alternate in their bids. The documents foresee the nomination of an Adjudicator whose role is to review and decide upon any matters of potential dispute between the parties where the parties, with or without the help of the Project Manager, have been unable to settle the matter amicably. The costs of the Adjudicator are shared equally by both parties. Arbitration is resorted to only if the parties fail to settle the dispute through the Adjudicator.

(iv) The General Conditions of Contract are based on the Model Form of International Contract for Process Plant Construction published by the Engineering Advancement Association of Japan (ENAA), and the Bank gratefully acknowledges the permission of ENAA to make use of and amend the Model Form for inclusion in these Standard Bidding Documents. These General Conditions meet the Bank’s requirements, and they shall be used without amendment unless specifically agreed with the Bank.

The ENAA conditions were drafted primarily with turnkey contracts in mind where the Contractor is responsible for each activity required for completion of the facilities, e.g., design, manufacture, delivery, installation, testing, commissioning, training, etc. However, these conditions have been adapted by the Bank for use for single responsibility contracts where some activities, such as parts of the preliminary design or site preparation works, are done by others.

(v) The time allowed for preparing and submitting bids should not be too short and should allow adequate time for bidders to properly study the Invitation, visit the site and prepare complete and responsive bids.

 I. Bidding Documents Issued by Employer Invitation for Bids Instructions to Bidders Bid Data Sheet General Conditions of Contract Special Conditions of Contract Technical Specifications and Drawings Sample Forms and Procedures  Eligibility for the Provision of Goods, Works and Services in Bank-Financed Procurement II. The Bid  Submitted by Bidder Cover Letter Bid Form Price Schedules Attachments: 1. Bid Security 2. Power of Attorney 3. Bidder’s Eligibility (Joint Venture, if any) 4. Eligibility of Facilities 5. Subcontractors proposed by Bidder (if any) 6. Deviations (if any) 7. Alternative Bids (if any) Technical Specifications and Drawings from the Bidder (if called for) III. Contract Documents  Issued by Employer & Submitted by Bidder Notification of Award Bid Form Price Schedules General Conditions of Contract Special Conditions of Contract Contract Agreement and Appendices 1. Terms and Procedures of Payment 2. Price Adjustment (if applicable) 3. Insurance Requirement 4. Time Schedule 5. List of Subcontractors 6. Scope of Works and Supply by Employer 7. List of Documents for Approval/Review 8. Functional Guarantees Forms and Procedures Technical Specifications & Drawings Any other documents agreed as forming a part of the Contract

Option B: Two Stage Bidding

Two Stage Bidding Procedure Flowchart

 I. Bidding Documents Issued by Employer II. The First Stage Bid Submitted by Bidder Invitation for Bids Instructions to BiddersBid Data Sheet General Conditions of Contract  Special Conditions of Contract Technical Specifications and Drawings Sample Forms and ProceduresEligibility for the Provision of Goods, Works and Services in Bank-Financed Procurement Cover Letter Bid Form-First StageAttachments: 1. Bidder’s Eligibility (Joint Venture, if any) 2. Facilities’ Eligibility 3. Subcontractors proposed by Bidder (if any) 4. Deviations and/or Alternative Bids Technical Specifications and Drawings from the Bidder (if called for)   III. Clarification Meeting & Invitation for Second Stage BidReminder of Clarification Meeting  ` IV. The Second Stage Bid Submitted by Bidder V. Contract Documents Cover Letter Memorandum of Clarification MeetingUpdated Technical Bid Bid Form-Second Stage Price Schedules Attachments: 1. Bid Security  2. Power of Attorney  3. Bidder’s Eligibility  4. Facilities’ Eligibility  5. Subcontractors proposed by Bidder (if any) Updated Technical Specifications and Drawings from the Bidder (in accord with Memorandum) Notification of Award Memorandum of Clarification MeetingBid Form Price Schedules General Conditions of Contract Special Conditions of Contract Contract Agreement and Appendixes 1. Terms and Procedures of Payment 2. Price Adjustment (if applicable) 3. Insurance Requirement 4. Time Schedule 5. List of Subcontractors 6. Scope of Works and Supply by Employer 7. List of Documents for Approval/Review 8. Functional Guarantees Forms and Procedures Technical Specifications & Drawings Any other documents agreed as forming a part of the Contract
http://www.unidroit.org/english/principles/contracts/main.htm

# International standard forms of contract

http://www.practicallaw.com/5-502-8478?q=&qp=&qo=&qe=

http://www.enaa.or.jp/EN/activities/model.html

Standard Bidding Document  Supply and Installation of Plant and Equipment

World Bank document index  Home > Projects > Procurement > Index of Standard Bi… > Document Index

http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:20062006~menuPK:84284~pagePK:84269~piPK:60001558~theSitePK:84266,00.html

## New Engineering Contract NEC

The NEC conditions of contract are made up of the following construction conditions:
The Engineering and Construction Contract (ECC)

• Option A: Priced contract with activity schedule
• Option B: Priced contract with bill of quantities
• Option C: Target contract with activity schedule
• Option D: Target contract with bill of quantities
• Option E: Cost reimbursable contract
• Option F: Management contract
• The Engineering and Construction Short Contract

• The Engineering and Construction Subcontract Contract

The primary differences between the options of the ECC are the mechanisms by which the contractor is reimbursed and encouraged to be cost effective. Therefore clauses that seek to remove barriers, and positively encourage the use of recycled and secondary aggregates, could be different in each case.

The guidance and example contract clauses will focus on ECC Options A, B and C. These options provide a framework to demonstrate how tender and contract clauses that encourage the use of recycled and secondary aggregates, can be integrated with the NEC conditions of contract.

Whilst the other options are not specifically considered, the guidance and example contract clauses set out below may still be drawn upon, to encourage the use of recycled and secondary aggregates.

As for the Engineering and Construction Short Contract and the Engineering and Construction Subcontract Contract, the flexibility of the NEC is such that any of options A to F can be incorporated as the drafter of the tender wishes.

More information on the original site itself: http://aggregain.wrap.org.uk/procurement/how_to_procure/model_clauses/nec_contracts/

The developer of the NEC the Institution of Civil Engineers UK   http://www.ice.org.uk/topics/lawandcontracts/NEC-and-ICE-contracts

Other site http://www.neccontract.com/

Comparison between FIDIC and NEC

(from http://en.allexperts.com/q/Construction-Law-914/2009/10/FIDIC-vs-NEC.htm accessed 14/11/11 )

Please read carefully from Mr. Agne Sandberg’s (Head of Legal Services) statements as follows,

5. COMPARISON BETWEEN THE SILVER BOOK AND THE ENAA MODEL FORM ON SOME ISSUES OF
PRINCIPLE INTEREST
I have chosen to look closer at certain matters of particular interest for turnkey contracts. These are;
Responsibility for errors in information provided by the Employer;
• Responsibility for errors in the Employer’s Requirements;
• Interference by the Employer in the performance by the Contractor;
Already at this stage I would like to point out that the issues chosen are not matters of allocation risk for events beyond the
Control of the Parties. If you would consider also the above issues to be matters of risk allocation, it is risks within the control of one
Of the parties I„e„ the Employer, to place that risk on the Contractor obviously contradicts the principle that the party that best
Can control a risk should carry it.
5.1 Responsibility for errors in information provided by the Employer. The Silver Book
It is obviously intended that the Employer shall not be responsible for errors or other incorrectness in information provided by him
To the Contractor,.
Sub-Clause 4.10 – Site Data
“The Employer shall have made available to the Contractor for his information, prior to the Base Date, all relevant data in the
Employer’s possession on hydrological and sub-surface conditions at the Site, including environmental aspects. The
Contractor shall be responsible for verifying and interpreting all such data,”
Sub-Clause 5.1 – General Design Obligations;
The Employer ………………………. shall not be deemed to have given any representation of accuracy or completeness of any
Information. Any information received by the Contractor, from the Employer or otherwise, shall not relieve the Contractor f om
his responsibility for the design of the Works,”
By Sub-Clause 4.12 it is confirmed that the Contractor will have to bear the consequences of incorrectness in the information
provided by the Employer, as the Contractor is deemed to have obtained all necessary information and to have foreseen all
difficulties.
Sub-Clause 4.12 – Unforeseeable Difficulties;
“The Contractor shall be deemed to have obtained all necessary information as to risks, contingencies and other
circumstances which may influence or affect the Works.. By signing the Contract, the Contractor accepts responsibility for
having foreseen all difficulties and costs of successfully completing the Works, The Contract Price shall not be adjusted to
take account of any unforeseen difficulties or costs, except as otherwise stated in the Contract,”
4
In other words, unless otherwise stated in Part II – Conditions of Particular Application, all data and information provided by
the Employer is used by the Contractor on his own risk..
ENAA Model Form
The Owner is responsible for the information to be provided by him during the course of the Contract. GC 10.1 – Owner’s
Responsibilities;
“The Owner shall ensure the correctness and exactitude of all information and/or data to be supplied by the Owner as described
in Appendix 9-3 (Scope of Works and Supply by the Owner) except when otherwise expressly stated in the Contract,”
It is also clear that the Owner is responsible for information submitted prior to signing Contract.
GC 20.1.1 – Specifications and Drawings;
“The Contractor shall be responsible for any discrepancies, errors or omissions in the specifications, drawings and other technical
documents prepared by it, whether such specifications, drawings and other documents have been approved by the Owner or
not, provided that such discrepancies, errors or omissions are not due to inaccurate information furnished in writing to the
Contractor by or on behalf of the Owner.”
The ENAA Model Form provides that the Contractor shall make a reasonable examination of information provided by the Owner, a
reasonable site investigation and examination of other data available.
GC 9.2 – Contractor’s Responsibilities;
“The Contractor confirms that it has entered into this Contract on the basis of a reasonable examination of the data relating
to the Works (including any data as to boring tests) provided by the Owner and of information which it could have
obtained from a visual inspection of the Site (if access thereto was available) and of other data readily available to it relating
to the Works as on the date of the Agreement”
Comments
Obviously the two conditions of contract have two opposite principles as concerns the Employer’s responsibility for
information provided by him. The ENAA Model Form has fully adopted the basic principle that normally applies for contracts, i.e.
that a party is responsible for that information provided by him is correct.. The Silver Book does not even have an option for the
situation where the Employer shall be responsible for certain information provided by him.
In larger contracts the Contractor can not obtain certain fundamental information or data from any other source than the Employer.
Consequently he should then not be held responsible if’ such information is incorrect.. The location of the Site, physical limits of
the Works, interfaces and for particular types of projects e.g.. quality of the fuel for a thermal power plant or water flow and
net head for a hydro power project, are certainly information that must be provided by the Employer and that he must retain
responsibility for.
For the purpose of this discussion, I consider that the Silver Book standpoint never can fully apply in any proper contract.
Even in the most pure BOT situation certain basic facts connected to the commercial foundation of the project can not be offloaded
to the turnkey contractor. In my opinion, not even in the situation where the Contractor also is shareholder in the
concession company. The wording of the Silver Book will therefore not apply to its full extent.. Also considering this, the
burden put on the Contractor is of course very heavy. The Contractor will as the collector and responsible for all information of
importance for execution of the Works have a knowledge of the project that the Employer will not have. This will be of
importance for the discussion on the Employer interference below..
5.2 Responsibility for errors in the Employer’s Requirements The
Silver Book
The form of tender normally states that the tenderer has based his tender on the documents included in invitation. The
Letter of Tender included in the Silver Book contain a statement that the tender has ascertained that the Conditions of
Contract, Employer’s Requirements do not contain any errors or other defects.
5
The conditions of contract shifts virtually all responsibility for the Employer’s Requirements to the Contractor. Sub-Clause
5.1 states that;
“The Contractor shall be responsible for the design of the Works and for the accuracy and completeness of the Employer’s
Requirements…”
and that
“The Employer shall not be responsible for any error, inaccuracy or omission of any kind in the Employer’s Requirements, and
shall not be deemed to have given any representation of accuracy or completeness of any information”
This does not sit well with the definition in Sub-Clause 1.1.1.3 which states that;
“the Employer’s Requirements” is the specification of the purpose, scope, and/or design and/or other technical
criteria.
and therefore the conditions does take a step back later in Sub-Clause 5.1 by providing that;
“However, the Employer shall be responsible for the correctness of the following portions of the Employer’s
Requirements:
(a) definition of intended purposes for the Works or any parts thereof,(b) criteria for testing/performance of
completed Works, and
(c) any other portions which are stated as being the responsibility of the Employer.”
ENAA Model Form
The Contractor shall perform the works in accordance with the Technical Specifications.

From this above comparison, I will be sure you will get good ideas. If, you need Mr. Agne Sandberg’s full comparison means, please send to me your personal email id. I will forward immediately. I think my answer will get full satisfaction to you. If, you fully satisfied please quote full mark to me.

Good resource on construction law in austrlia

http://www.mtecc.com.au/resources/publications-papers

Good write up on FIDIC                 http://www.mtecc.com.au/uploads//papers/Shnookal,_Toby.pdf