enables coordination and synchronization with what we calls “znodes”,
which are presented as a directory tree and resemble the file path names we
would see in a Unix file system. Znodes are capable of storing data but not
much to speak of — currently less than 1 MB by default. The concept here is
that Zookeeper stores znodes in memory and that these memory-based znodes enables
quick client access for coordination, status, and other important functions needed
by distributed applications like HBase. Zookeeper replicates (make copies)
znodes across the ensemble so if servers fail, the znode data will still be available
as long as a majority quorum of servers is still up and running.
next crucial Zookeeper concept concerns about how znode reads (versus writes)
are managed. Every Zookeeper server can manage reads from a client, including
the leader, however only the leader issues atomic znode writes — writes which
are either completely succeed or completely fail. When a znode write request comes
at the leader node, the leader broadcasts the write request to its following nodes
and then waits for a majority of followers to acknowledge this znode write
complete. After the acknowledgement, the leader issues the znode write itself
and then reports the successful completion status to the client
enables some very powerful and flexible guarantees. When a Zookeeper client (like
a HBase RegionServer) writes or reads a znode, this operation is atomic. It
either completely succeeds or completely fails — there doesn’t exist any
partial reads or writes. No other competing client can lead the read or write operation
to fail. Furthermore, a znode has an access control lists (ACL) associated
with it for security purposes, which supports versions, timestamps and
notification to clients when it modifies.
replicates znodes across the ensemble so if a server fail, the znode data is still
available as long as a majority quorum of servers is still up and running. This
implies that writes to any znode from any Zookeeper server must be propagated
across the ensemble. The Zookeeper leader handles this operation.
znode write mechanism can result followers to fall behind the leader for short
periods. Zookeeper resolves this potential problem by providing a
synchronization command. Users that cannot tolerate this temporary lack of synchronization
within the Zookeeper cluster may decide to issue a sync command before reading
znode world, we are going to come across what looks like the Unix-style pathnames.
(Typically they begin with /hbase.) These pathnames, which are a subset of the
znodes in the Zookeeper system created by HBase, are described in this list:
master: Holds the name of
the primary MasterServer,
hbaseid: Holds the cluster’s
to the RegionServer holding the -ROOT- table),
Something called /hbase/rs.
now we may wonder what’sup with this rather vaguely defined /hbase/rs.
In the previous post (HBase Architecture: Master Server Part 4),
we describe the various operations of the MasterServer and mention that
Zookeeper notifies the MasterServer whenever a RegionServer fails. Now we help
you take a closer look at how the process actually works in HBase — and we would
be right to assume that it has something to do with /hbase/rs. Zookeeper uses
its watches mechanism to notify clients whenever a znode is created, accessed,
or changed in some way. The MasterServers are Zookeeper clients as well as the
RegionServers and can leverage these znode watches. When a RegionServer comes
online in the HBase system, it connects to the Zookeeper ensemble and creates
its own unique ephemeral znode under the znode pathname /hbase/rs. At the same time,
the Zookeeper system establishes a session with the RegionServer and monitors
the session for events. If the RegionServer breaks the session for whatever
reason (by failing to send a heartbeat ping, for example), the ephemeral znode
that it created is deleted. The action of deleting the RegionServer’s child
node under /hbase/rs will cause the MasterServer to be notified so that it can
initiate RegionServer failover. This notification is accomplished by way of a
watch that the MasterServer sets up on the /hbase/rs znode.