Machine Learning with Spark Nick Pentreath instant download
Machine Learning with Spark Nick Pentreath instant download
download
https://textbookfull.com/product/machine-learning-with-spark-
nick-pentreath/
https://textbookfull.com/product/mastering-apache-
spark-2-x-scale-your-machine-learning-and-deep-learning-systems-
with-sparkml-deeplearning4j-and-h2o-2nd-edition-romeo-kienzler/
https://textbookfull.com/product/predictive-analytics-with-
microsoft-azure-machine-learning-barga/
https://textbookfull.com/product/machine-learning-with-
tensorflow-2nd-edition-chris-a-mattmann/
https://textbookfull.com/product/advanced-analytics-with-spark-
patterns-for-learning-from-data-at-scale-2nd-edition-sandy-ryza/
Natural Language Processing with Spark NLP Learning to
Understand Text at Scale 1st Edition Alex Thomas
https://textbookfull.com/product/natural-language-processing-
with-spark-nlp-learning-to-understand-text-at-scale-1st-edition-
alex-thomas/
https://textbookfull.com/product/advanced-data-analytics-using-
python-with-machine-learning-deep-learning-and-nlp-examples-
mukhopadhyay/
https://textbookfull.com/product/applied-machine-learning-for-
health-and-fitness-a-practical-guide-to-machine-learning-with-
deep-vision-sensors-and-iot-kevin-ashley/
https://textbookfull.com/product/stream-processing-with-apache-
spark-mastering-structured-streaming-and-spark-streaming-1st-
edition-gerard-maas/
https://textbookfull.com/product/hands-on-machine-learning-with-
scikit-learn-and-tensorflow-aurelien-geron/
Preface
In recent years, the volume of data being collected, stored, and
analyzed has exploded, in particular in relation to activity on
the Web and mobile devices, as well as data from the physical
world collected via sensor networks. While large-scale data
storage, processing, analysis, and modeling were previously the
domain of the largest institutions, such as Google, Yahoo!,
Facebook, Twitter, and Salesforce, increasingly, many
organizations are being faced with the challenge of how to
handle a massive amount of data.
Chapter 12, Pipeline APIs for Spark ML, provides a uniform set
of APIs that are built on top of Data Frames and help the user
to create and tune machine learning pipelines.
What you need for this
book
Throughout this book, we assume that you have some basic
experience with programming in Scala or Python and have
some basic knowledge of machine learning, statistics, and data
analysis.
Who this book is for
This book is aimed at entry-level to intermediate data
scientists, data analysts, software engineers, and practitioners
involved in machine learning or data mining with an interest in
large-scale machine learning approaches, but who are not
necessarily familiar with Spark. You may have some experience
of statistics or machine learning software (perhaps including
MATLAB, scikit-learn, Mahout, R, Weka, and so on) or
distributed systems (including some exposure to Hadoop).
Conventions
In this book, you will find a number of text styles that
distinguish between different kinds of information. Here are
some examples of these styles and an explanation of their
meaning.
Once the file is downloaded, please make sure that you unzip or
extract the folder using the latest version of:
The code bundle for the book is also hosted on GitHub at http
s://github.com/PacktPublishing/Machine-Learning-with-Spark-Second-Edit
ion. We also have other code bundles from our rich catalog of
books and videos available at https://github.com/PacktPublishing/.
Check them out!
Downloading the color
images of this book
We also provide you with a PDF file that has color images of
the screenshots/diagrams used in this book. The color images
will help you better understand the changes in the output. You
can download this file from https://www.packtpub.com/sites/default/
files/downloads/MachineLearningwithSparkSecondEdition_ColorImages.pdf.
Errata
Although we have taken every care to ensure the accuracy of
our content, mistakes do happen. If you find a mistake in one
of our books-maybe a mistake in the text or the code-we would
be grateful if you could report this to us. By doing so, you can
save other readers from frustration and help us improve
subsequent versions of this book. If you find any errata, please
report them by visiting http://www.packtpub.com/submit-errata,
selecting your book, clicking on the Errata Submission Form
link, and entering the details of your errata. Once your errata
are verified, your submission will be accepted and the errata
will be uploaded to our website or added to any list of existing
errata under the Errata section of that title.
http://spark.apache.org/community.html
http://spark.apache.org/community.html#history
Source: https://amplab.cs.berkeley.edu/wp-content/uploads/2011/11/spark-lr.png
The standalone local mode, where all Spark processes are run
within the same Java Virtual Machine (JVM) process
...
16/11/24 14:41:58 INFO Executor: Finished task 99.0 in stage 0.0
(TID 99). 872 bytes result sent to driver
16/11/24 14:41:58 INFO TaskSetManager: Finished task 99.0 in stage
0.0 (TID 99) in 59 ms on localhost (100/100)
16/11/24 14:41:58 INFO DAGScheduler: ResultStage 0 (reduce at
SparkPi.scala:38) finished in 1.988 s
16/11/24 14:41:58 INFO TaskSchedulerImpl: Removed TaskSet 0.0,
whose tasks have all completed, from pool
16/11/24 14:41:58 INFO DAGScheduler: Job 0 finished: reduce at
SparkPi.scala:38, took 2.235920 s
Pi is roughly 3.1409527140952713
The preceding command calls class
org.apache.spark.examples.SparkPi class.
The ocean pilots and deep sea divers of New York have one thing in
common; both object to taking apprentices, and in the case of the
former, at least, there is good reason for this, since they have been,
for generations, the aristocrats of their calling. The pilots who sail
out of Sandy Hook are no hardier than their rugged and fearless
fellows of the North Sea, but they subject themselves to greater
dangers by their long cruises, and rough, indeed, must be the
weather that can keep them in port. They cruise night and day, in
search of incoming craft; their torches' flare lights up the snow and
sleet of winter storms and contends with the darkness of summer
fogs; and they speak and board in all sorts of weather and at all
seasons the fleet liners that cross the western ocean in less than a
week. And these pilots of the New York and New Jersey shores are a
revelation to the tourist, who, having never heard of them, sees
them for the first time. The latter, in most cases, expects to watch a
rough-and-ready sort of fellow in homespun, with a swaggering air
and a boisterous manner, climb from the pilot's yawl up the black
hull's towering side. Instead, he sees a man of modest and pleasing
address, about whom there is little to indicate his calling, and much
that bespeaks the merchant or clerk one meets of a morning on
lower Broadway. There was a time when our pilots indulged in the
luxury of a high silk hat when boarding vessels in sunny weather, but
they are not so fastidious nowadays, and use derbies instead.
Prosperous as a class, the pilots of New York pay dearly for their
prosperity by the most arduous sea labor. Since 1853 more than
thirty-five boats have been sunk and wrecked in various ways, and
twice that number of pilots have lost their lives. There are at the
present time upward of 160 pilots cruising from the port of New
York. They are subject to the supervision of a pilots' commission of
five members, named by the Governor of New York, and each pilot is
appointed after a long and severe apprenticeship. He must first
serve, boy and man, before the mast until he masters every problem
in the management of every form of rig. Then he must contrive to
obtain the position of boat-keeper or pilot's mate. In that capacity he
must serve three full years before he can be admitted for his
examination for a license. After this he must pass a most rigid
examination on all points of seamanship and navigation before the
Board of Pilot Commissioners, and show complete and exact
knowledge of the tides, rips and sands and all other phenomena for
many miles out from the piers of the East and North Rivers.
But even after the candidate has received his license, he is
sometimes forced to wait years, until some pilot happens to die and
leave a vacancy for him. The first year of pilotage he is granted a
license to pilot vessels drawing less than sixteen feet. If he gives
satisfaction, the following year he is permitted to take charge of
vessels drawing eighteen feet. If he passes a satisfactory
examination the third year, he then receives a full license, entitling
him to pilot vessels of any draught, and is then first called a branch
or full pilot. On receiving his license, the pilot must give bonds for
the proper discharge of his duty, and he is liable to heavy fines if he
declines to fill a vacancy or board a vessel making signals for a pilot.
Pilots are paid for their work by the foot, the charges varying
according to the draught. For a ship drawing from twenty-one to
twenty-eight feet they receive $4.88 a foot, and for one drawing six
to thirteen and one-half feet $2.78 a foot, these rates being slightly
increased in winter.
A cruise on a New York pilot-boat, however brief, is an experience
sure to be remembered. When a pilot-boat starts out on a hunt for
ships, it is decided in what order its half-dozen pilots shall take the
prizes, and the man who is to board the first one is placed in
command. The other pilots, meanwhile, take their ease as best suits
their taste, the seaman's work being done by a crew of sailors hired
for the purpose. One pilot, however, is always on the lookout for
sails, and a landsman is compelled to marvel at the certainty with
which these ocean scouts discharge the task of sighting vessels, for
often they are able to tell the name of a steamship before
unaccustomed eyes can discern aught but a waste of waters and a
wide expanse of sky. Still, a part of this skill may be due to the fact
that pilots are always posted before going out as to what vessels are
expected, and from what direction they are coming, the watch being
made all the keener by the fact that the bigger the ship the bigger is
the pilot's pay. A ship, moreover, must take a pilot going out from
the same boat that furnishes the pilot going into port, while if a
captain refuses a pilot he must pay full pilotage, and thus contribute
his tithe to the support of the system. This latter rule seems, at first
glance, a curious provision, but it is defended on the ground that
without it the business would not be remunerative enough for really
competent men to engage in it, and that with unskilled pilots the
annual losses would be greatly in excess of what they are at
present.
When a ship is sighted by daylight, a long blue burgee is hoisted to
the peak of the pilot-boat, which means, "Do you want a pilot?" If
there is no responsive signal, it is taken for granted that the answer
is "Yes," but if a jack is hoisted the watchers know that the vessel
has already been boarded by a pilot from some boat that has sailed
farther away from port in the hunt for a ship. When a ship is sighted
at night she is signalled by means of a torch charged with benzine
and giving forth an intense light. Seen from the other vessel the
effect is startling, the white light illuminating every sail and spar of
the pilot-boat, so that it stands out, its number clearly visible upon
the mainsail, a gray specter against the night's background.
Should the answering signal be favorable, there follows a scene of
great excitement on the deck of the pilot-boat. At first sight of the
ship, the pilot due to take the prize dives down to the cabin, sheds
his working clothes and dons a suit of sober black, and by the time it
is known he is wanted, he is ready to be transferred to his charge.
Taking on a pilot is not without its perils. The yawl nearly always
pitches and tumbles in most uncomfortable fashion, while the ship is
rarely if ever brought to a full stop, and the pilot, watching his
chance, must grasp the rope ladder let down its side, and scramble
aboard as best he can. Sometimes he gets a ducking, and if the
weather is tempestuous he is pretty certain to be drenched, but for
that he cares not a jot, and he is sure to show a smiling face to
captain and passengers when finally he sets foot on deck. Dropping
a pilot from an outgoing vessel is often more hazardous, especially
in stormy weather, than his transfer the other way. Then he must
descend the rope ladder and jump for the boat in the nick of time,
for to miscalculate in the least the position of the little shell means a
ducking almost certainly, and possibly a watery grave.
PILOT SIGNALING A VESSEL
A peril, however, more feared by pilots than the one I have been
describing, is the dreaded lee shore; and with reason, as a story told
by a veteran ocean pathfinder will show. On a still afternoon in
midsummer the crew of a pilot-boat sighted a ship off Fire Island,
some five miles away. In the dead calm prevailing the only way to
board her was to row over the distance. There would be little danger
in doing this if the wind did not spring up and the ship sail away, so
the yawl was lowered and headed for the distant merchantman. But
as night was closing in, and ere the yawl had come within hailing
distance of the ship, of a sudden the breeze sprang up, and the
vessel making sail, glided slowly over the horizon line. The breeze
grew into a gale, and in the gathering storm and gloom the men
could no longer discern the whereabouts of the pilot-boat. Nor, there
being no compass on board the yawl, could they determine the
direction in which they were being blown. The nearest land was
miles away and the only thing that could be done was to keep the
boat's head to the wind and wait. Thus the minutes lengthened into
hours. Toward dawn, when the night was darkest, they heard the
thunder of surf on the reefs, and a little later felt the yawl lifted up
on the crest of a mighty breaker rushing swiftly toward the land.
There was a deafening roar, a crash, a whirl, and a torrent of foam.
In a twinkling the boat was capsized and the poor fellows were
struggling in the surf. One struck a rock and was killed. The others,
freed from the receding wave, ran up the beach, and by digging
their hands into the sand to escape the deadly undertow, finally got
ashore, drenched and exhausted.
In the main, however, the system I have been describing has now
become a thing of the past. Potent causes have contributed to this
result. Formerly pilot-boats had no particular stations assigned to
them, and boats have been known to cruise as far north as Sable
Island, a distance of six hundred miles, in order to get steamers
taking the northern courses. In the same way pilot-boats cruised
long distances to the southward and straight out to sea to meet the
incoming steamers and sailing vessels. Thus, unrestrained in its
movements and left to seek out its own salvation, each boat sought
to outdo the other in securing work, and all sorts of strategic devices
were brought into play in order to first gain the side of an incoming
vessel. Pilots took advantage of fog and night in order to slip by a
rival, while jockeying for winds and position was indulged in to an
extent that would be counted extraordinary in a yacht race.
Competition, however, cut down earnings to such an extent that
there came a time when many of the boats were no longer able to
pay expenses. Then it was that some of the long-headed among the
pilots, casting about for a remedy for this evil, came to the
conclusion that one steam pilot-boat would be able to do the work of
three or four sailboats. It was accordingly decided some years ago
that steamboats should gradually replace the existing fleet of sail.
With this innovation came restrictions regulating the cruising
grounds of the boats. Instead of cruising about indiscriminately as
formerly, each boat is now assigned a certain beat. An imaginary arc
has been described extending from Barnegat to Fire Island, a
distance of seventy-five miles, and all pilot boats are expected to
confine themselves within this line. Four pilot-boats patrol this line,
each covering a beat of about nineteen miles. Inside of the circle are
stationed two more pilot-boats, while still further in is a boat known
as the inner pilot-boat. Just off the bar another boat is stationed to
receive the pilots dropped by outward-bound vessels. When a boat
in the outer circle becomes unmanned or disabled, a boat from the
inner circle takes its place, while a reserve boat occupies the beat
left vacant on the inner circle. In this way all the beats are
constantly patrolled in an efficient and economical way. Each pilot
takes his turn at the service, and is on board a boat cruising on the
stations three days in seven, a moving contrast to the offshore
service of other years, when a boat and crew were frequently
compelled to remain at sea for weeks at a time.
Indeed, under the new system of pilotage, battles with cross-seas
and gales and exposures to snow, cold and sleet, while cruising for
vessels hundreds of miles off coast, are fast becoming things of the
past, and for stories of collisions, wrecks, narrow escapes and
strange mishaps, one must now hark back to the records of former
days. Here, however, he is sure to encounter many a tale that
quickens the pulse and stirs the blood. Take the case of the
Columbia, run down by the steamship Alaska, off Fire Island. When
the Alaska was sighted, the pilot-boat was head-reaching to the
north on the port tack. The wind was blowing a gale from the
northwest, and an ugly sea was running, with the weather clear, but
cold. She plunged deeply into the heavy sea, and heeled to the force
of the wind until her lee rail was awash. The wind whipped off the
top of the waves and filled the air with spray. When the steamship
sighted the boat off Fire Island, her course was changed to make a
lee for the boat's yawl. She seemed to stop when the yawl was
launched and two men and a pilot went over the side of the boat
and dropped into her, but ere the yawl had fairly started on her way
the liner, of a sudden, and without warning, forged ahead. The surge
from the port bow of the Alaska, as she pitched into a big wave,
capsized the boat, and threw the men into the water. Before
anything could be done to save them the bows of the steamship
rose and fell again, and, hitting the pilot-boat, cut it in two and
crushed the decks and beams to bits, the broken timbers being
swept under the bows and along the sides as the steamship again
forged ahead and passed over the spot. Not a man on the Columbia
was saved.
The Sandy Hook pilot, however, never quails in the face of danger or
even death, as was proved at the stranding of the packet boat, John
Minturn, almost within a stone's throw of the New Jersey beach
during a frightful hurricane in February, 1846. There were fifty-one
souls on board the Minturn, and of that number only thirteen
escaped to tell the story of that fearful night. Its hero, according to
the evidences of all, was Pilot Thomas Freeborn, who to the very last
struggled manfully to succor the hapless women and children who
clung to the deck around him. It was bitter cold, and every wave
that washed over the stranded ship left its coating of ice on deck,
rigging, passengers and crew. Freeborn and brave Captain Stark,
who was forced to see his wife and children freeze to death without
being able to render them assistance, gave up their own clothing in
a vain attempt to protect the weaker sufferers, and when days
afterward the pilot's body was found washed up on the beach it was
almost naked, while that of a woman, which lay near-by, was
carefully wrapped in his pea-jacket.
It has been three-score years since the wreck of the Minturn, but in
every year since then there has been numbered among the
members of the Sandy Hook Pilot's Association scores of hardy men,
who, should need come to them, stood ready to risk their lives and
die as bravely as did Thomas Freeborn. Pilot Henry Devere proved
that he had the same heroic fiber in his makeup when he sailed in
the James Funck, before the Civil War. A brig under shortened sails
was sighted one day, and when the yawl of the pilot-boat drew
alongside, Devere hailed a boy at the wheel. The boy seemed to be
stupefied, and the pilot was obliged to hail him several times before
he started up, leaned forward into the companionway, and called
feebly to somebody below. Then a gaunt man came on deck and
said that the crew had been stricken by fever. Most people in the
face of a menace of this sort would have turned back, but Devere
was not that kind of man. Instead, he went on board, and, with the
help of the mate, headed the vessel toward Sandy Hook. The
captain was ill in his stateroom. The body of a dead sailor found on
deck was tied in mosquito netting and dropped overboard. The boy
died in the lower bay, and the captain off the Battery, leaving the
mate as the sole survivor of the crew. The pilot helped to furl the
sails and make the lines fast, and only left the stricken vessel when
she had reached her moorings.
The stranding of the Jesse Carll in 1889, illustrates another of the
dangers with which pilots sometimes have to contend. The boat,
having discharged one of her five pilots, was standing off shore near
Fire Island, when she began to feel the force of an advancing
southern cyclone, and early in the evening was in what sailors call
"nasty weather." At midnight a violent thunder-storm burst
overhead, and the increasing wind raised a furious sea, but Pilot
Gideon Mapes, in charge of the vessel, had her under double-reefed
sails, and standing up against the wind and waves in fine shape.
Then came a deluge of rain, and the wind increased to hurricane
force. Soon a thick mist covered the water and shut out everything
in sight. The boat reached off and on, expecting to keep out of shoal
water, but all efforts failed. Her signals of distress were seen by the
life-saving crew on the beach, and before daylight the ten men on
board were taken ashore in boats. When morning came an effort
was made to pull the boat off, but as she shifted into deeper water
she filled, a hole having been made in her bottom. Then the pilots
abandoned her, but she was raised and repaired a few weeks later.
Stories like these are what the pilots tell in their idle hours.
Searching for them at such a time, one is most likely to find them at
the Pilots' Club, a flourishing social organization, which has roomy
quarters just under the roof of a big office building within hailing
distance of the Battery. Here at all hours of the day a score or more
of pilots are sure to be sitting about spinning yarns, playing cards
and checkers and reading the newspapers and magazines. Their
well-furnished clubrooms contain a great number of precious curios
—relics from all quarters of the globe. There are firearms of curious
antique pattern; autograph letters by such famous sea-dogs as
Macdonough and Porter; a tiny chest of drawers carved from one of
the timbers of John Paul Jones' ship, the Bon Homme Richard; a
portrait of Washington by Stuart, surrounded by two large American
flags, and a model of the pilot-boat Stingaree, which was built in
1810, and was one of the most famous crafts of her day.
This model shows that the years have wrought great changes in the
building and rigging of pilot-boats. In old times the boats simply
carried mainsail, foresail, and forestaysail and jib. They had no
foretopmast, and on their maintopmast carried a flying gaff-topsail,
which was hoisted from the deck. Now the boats have both fore and
maintopmasts, and each carries a mainsail, foresail, forestaysail, jib,
jib-topsail, maintopsail and staysail and fore and main standing-gaff
topsails, which give them an immense spread of sail, compared with
that used by the boats of earlier times. A schooner-rigged pilot-boat
costs from $15,000 to $16,000. That was about the cost of the
Caldwell H. Colt, a good example of the typical pilot-boat. She is
eighty-five feet long with twenty-one feet beam, 61.43 tons, custom-
house register, and a rig as trim and jaunty as that of an ordinary
yacht. The pride, however, at present writing, of the New York Sandy
Hook fleet is the New York, built of steel, propelled by steam, and
able to stand as much buffeting in cyclonic seas as the stanchest of
the liners. She was built on the Delaware from designs by A. Cary
Smith, is 155 feet long, 28 feet beam, 19 feet 7 inches deep, and is
driven by a compound surface-condensing engine of 100 horse-
power. Her pole masts are of steel, and she spreads on them enough
canvas to steady her. The New York has accommodations for twenty-
four pilots, who fare more luxuriously than they ever did on any of
the old sailing craft. They have a smoking-room in a separate steel
deckhouse, aft of the engine-room, fitted up like a similar room on
an ocean steamship, while the lifeboats in which they leave the New
York to board incoming vessels are hoisted and lowered by a steam
derrick in less than a minute. It is intended that in a few years the
entire fleet shall be made up of vessels equal if not superior to the
New York.
CHAPTER VII
THE DEEP-SEA DIVER
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
textbookfull.com