public class WorkerStorage extends Object
Modifier and Type | Class and Description |
---|---|
class |
WorkerStorage.CheckpointThread
The CheckpointThread, used to checkpoint the files belong to the worker.
|
Constructor and Description |
---|
WorkerStorage(InetSocketAddress masterAddress,
ExecutorService executorService)
Main logic behind the worker process.
|
Modifier and Type | Method and Description |
---|---|
void |
addCheckpoint(long userId,
int fileId)
Add the checkpoint information of a file.
|
boolean |
asyncCheckpoint(int fileId)
Notify the worker to checkpoint the file asynchronously.
|
void |
cacheBlock(long userId,
long blockId)
Notify the worker the block is cached.
|
void |
cancelBlock(long userId,
long blockId)
Cancel the block which is being written by some user
|
void |
checkStatus()
Check worker's status.
|
void |
freeBlocks(List<Long> blockIds)
Remove blocks from Tachyon cache space.
|
StorageDir |
getStorageDirByBlockId(long blockId)
Get StorageDir which contains specified block
|
String |
getUfsOrphansFolder() |
String |
getUserUfsTempFolder(long userId)
Get the user temporary folder in the under file system of the specified user.
|
Command |
heartbeat()
Heartbeat with the TachyonMaster.
|
void |
initialize(NetAddress address) |
void |
initializeStorageTier()
Initialize StorageTiers on current WorkerStorage
|
StorageDir |
lockBlock(long blockId,
long userId)
Lock the block by some user
Used internally to make sure blocks are unmodified, but also used in
TachyonFS for caching blocks locally for users. |
boolean |
promoteBlock(long blockId)
If the block is not on top StorageTier, promote block to top StorageTier
|
void |
register()
Register this TachyonWorker to the TachyonMaster
|
String |
requestBlockLocation(long userId,
long blockId,
long initialBytes)
Get temporary file path for some block, it is used to choose appropriate StorageDir for some
block file with specified initial size.
|
boolean |
requestSpace(long userId,
long blockId,
long requestBytes)
Request space from the specified StorageDir, it is used for requesting space for the block
which is partially written in some StorageDir
|
void |
resetMasterClient()
Set a new MasterClient and connect to it.
|
void |
stop()
Disconnect to the Master.
|
boolean |
unlockBlock(long blockId,
long userId)
Unlock the block
Used internally to make sure blocks are unmodified, but also used in
TachyonFS for cacheing blocks locally for users. |
void |
userHeartbeat(long userId)
Handle the user's heartbeat.
|
public WorkerStorage(InetSocketAddress masterAddress, ExecutorService executorService)
initialize(tachyon.thrift.NetAddress)
must be called.masterAddress
- The TachyonMaster's addressexecutorService
- public void initialize(NetAddress address)
public void addCheckpoint(long userId, int fileId) throws FileDoesNotExistException, SuspectedFileSizeException, FailedToCheckpointException, BlockInfoException, IOException
userId
.
This method is normally triggered from FileOutStream.close()
if and only
if WriteType.isThrough()
is true. The current implementation of
checkpointing is that through WriteType
operations write to
UnderFileSystem
on the client's write path, but under a user temp directory
(temp directory is defined in the worker as getUserUfsTempFolder(long)
).userId
- The user id of the client who send the notificationfileId
- The id of the checkpointed fileFileDoesNotExistException
SuspectedFileSizeException
FailedToCheckpointException
BlockInfoException
IOException
public boolean asyncCheckpoint(int fileId) throws IOException
fileId
- The id of the fileIOException
public void cacheBlock(long userId, long blockId) throws FileDoesNotExistException, SuspectedFileSizeException, BlockInfoException, IOException
TachyonFS.cacheBlock(long)
which is
only ever called from BlockOutStream.close()
(though its a public api so
anyone could call it). There are a few interesting preconditions for this to work.
1) Client process writes to files locally under a tachyon defined temp directory. 2) Worker
process is on the same node as the client 3) Client is talking to the local worker directly
If all conditions are true, then and only then can this method ever be called; all operations
work on local files.userId
- The user id of the client who send the notificationblockId
- The id of the blockFileDoesNotExistException
SuspectedFileSizeException
BlockInfoException
IOException
public void cancelBlock(long userId, long blockId)
userId
- The id of the user who wants to cancel the blockblockId
- The id of the block that is cancelledpublic void checkStatus()
It finds the timeout users and cleans them up.
public void freeBlocks(List<Long> blockIds)
Command
with type CommandType.Free
blockIds
- The id list of blocks to be removed.public StorageDir getStorageDirByBlockId(long blockId)
blockId
- the id of the blockpublic String getUfsOrphansFolder()
public String getUserUfsTempFolder(long userId)
Users.getUserUfsTempFolder(long)
, and as such
should be referentially transparent with Users.getUserUfsTempFolder(long)
. In the
context of this
, this call will output the result of path concat of
mUfsWorkerFolder
with the provided userId.
This temp folder generated lives inside the UnderFileSystem
, and as such, will
be stored remotely, most likely on disk.userId
- The id of the userpublic Command heartbeat() throws IOException
IOException
public void initializeStorageTier() throws IOException
IOException
public StorageDir lockBlock(long blockId, long userId)
TachyonFS
for caching blocks locally for users. When a user tries to
read a block (TachyonFile.readByteBuffer(int)
()}), the client will
attempt to cache the block on the local users's node, while the user is reading from the local
block, the given block is locked and unlocked once read.blockId
- The id of the blockuserId
- The id of the user who locks the blockpublic boolean promoteBlock(long blockId)
blockId
- the id of the blockpublic void register()
public String requestBlockLocation(long userId, long blockId, long initialBytes) throws OutOfSpaceException, FileAlreadyExistException
userId
- the id of the user who wants to write the fileblockId
- the id of the blockinitialBytes
- the initial size allocated for the blockOutOfSpaceException
FileAlreadyExistException
public boolean requestSpace(long userId, long blockId, long requestBytes) throws FileDoesNotExistException
userId
- The id of the user who send the requestblockId
- The id of the block that the space is allocated forrequestBytes
- The requested space size, in bytesFileDoesNotExistException
public void resetMasterClient()
public void stop()
public boolean unlockBlock(long blockId, long userId)
TachyonFS
for cacheing blocks locally for users. When a user tries to
read a block (TachyonFile.readByteBuffer(int)
), the client will attempt
to cache the block on the local users's node, while the user is reading from the local block,
the given block is locked and unlocked once read.blockId
- The id of the blockuserId
- The id of the user who unlocks the blockpublic void userHeartbeat(long userId)
userId
- The id of the userCopyright © 2015. All Rights Reserved.