Patent US10257268

Distributed peer-to-peer data center management
[+] Show Details

US10257268

1. A method of managing rack-mounted computing devices in a data center with a distributed peer-to-peer management system, the method comprising: (17) (7)

determining roles of data-center management computing devices in a distributed peer-to-peer data-center management system;

receiving, via an out-of-band network, a data-center management command at a given data-center management computing device;

distributing, based on at least some of the roles, via the out-of-band network, the data-center management command from the given data-center management computing device to a first plurality of other data-center management computing devices; and

for each of the first plurality of other data-center management computing devices, distributing, based at least some of the roles, via the out-of-band network, the data-center management command to a respective second plurality of other data-center management computing devices, wherein: the given data-center management computing device, the first plurality of other data-center management computing devices, and the respective second pluralities of other data-center management computing devices are each configured to manage a respective plurality of rack-mounted computing devices;

each of the pluralities of respective rack-mounted computing devices executes an in-band application that communicates on an in-band network different from the out-of-band network; and

the data-center management command is a command to update or otherwise configure a microkernel, a virtual machine, a container, an operating system, or an application executing on one or more data-center management computing devices;

determining, with at least one of the first plurality of other data-center management computing devices, that the given data-center management computing device has failed;

electing, with at least one of the first plurality of other data-center management computing devices, one of the first plurality of other data-center management computing devices to serve as a replacement to the given data-center management computing device in at least some roles filled by the given data-center management computing device; and

distributing a result of the election to each the first plurality of other data-center management computing devices.

2. The method of claim 1, wherein: (1) (1)

determining roles of data-center management computing devices in a distributed peer-to-peer data-center management system comprises: selecting a primary data-center management computing device; and

selecting a plurality of leader data-center management computing devices by concurrently executing a plurality of respective instances of a distributed algorithm by which the respective leader is determined by consensus, each instance being executed by one of the second pluralities of other data-center management computing devices to select a respective leader for the respective one of the second pluralities of other data-center management computing devices.

3. The method of claim 2, wherein selecting a plurality of leader data-center management computing devices by concurrently executing a plurality of respective instances of a distributed algorithm by which the respective leader is determined by consensus comprises: (0) (7)

detecting with a first one of the data-center management computing devices that a signal has not been received from a leader within a threshold duration of time;

incrementing or decrementing a count with the with first one of the data-center management computing devices;

sending, with the first one of the data-center management computing devices, a request to vote for a new leader along with the incremented or decremented count to other data-center management computing devices in the same second plurality of other data-center management computing devices;

receiving the request with a second one of the data-center management computing devices;

determining, with the second one of the data-center management computing devices, to vote for the first one of the data-center management computing devices, at least in part, by: determining that the second one of the data-center management computing devices has not yet submitted a vote associated with the incremented or decremented count; and

determining that the request is the first request received by the second one of the data-center management computing devices in a given round of voting;

sending, with the second one of the data-center management computing devices, the vote to other data-center management computing devices in the same second plurality of other data-center management computing devices; and

selecting a leader based on amounts of votes received by candidate data-center management computing devices.

4. The method of claim 1, wherein: (1) (1)

determining roles comprises determining roles in a peer-to-peer distributed application having a structured topology.

5. The method of claim 4, wherein: (0) (1)

6. The method of claim 1, wherein: (0) (1)

7. The method of claim 1, comprising: (0) (1)

9. The method of claim 1, wherein: (0) (2)

10. The method of claim 1, wherein: (0) (2)

12. The method of claim 1, wherein: (0) (1)

13. The method of claim 1, wherein: (0) (1)

14. The method of claim 1, wherein: (0) (1)

15. The method of claim 1, wherein: (0) (1)

16. The method of claim 1, comprising implementing control and updates of rack controllers at the rack-level, in a distributed, concurrent routine. (0)

17. The method of claim 1, wherein for each of the first plurality of other data-center management computing devices, distributing, based at least some of the roles, via the out-of-band network, the data-center management command to a respective second plurality of other data-center management computing devices comprises: (0) (2)

18. The method of claim 1, comprising: (0) (1)

19. The method of claim 1, wherein determining roles of data-center management computing devices in a distributed peer-to-peer data-center management system comprises: (0) (1)

20. The method of claim 1, comprising: (0) (1)

21. The method of claim 1, wherein determining roles of data-center management computing devices in a distributed peer-to-peer data-center management system comprises: (0) (1)

22. The method of claim 1, comprising: (0) (1)

23. A system, comprising: (14) (1)

a plurality of racks, each rack comprising: a plurality of rack-mounted computing devices, each of the rack-mounted computing devices configured to execute an in-band application that communicates on an in-band network different from an out-of-band network; and

a medium storing instructions configured to instantiate a rack controller coupled to each of the rack-mounted computing devices, wherein each rack controller is configured to effectuate operations comprising: determining roles of the rack controllers in a distributed peer-to-peer data-center management system;

receiving, via the out-of-band network, a data-center management command at a first rack controller;

distributing, based on at least some of the roles, via the out-of-band network, the data-center management command from the first rack controller to a first plurality of other rack controllers;

for each of the first plurality of other rack controllers, distributing, based at least some of the roles, via the out-of-band network, the data-center management command to a respective second plurality of other rack controllers, wherein the first rack controller, the first plurality of other rack controllers, and the respective second pluralities of other rack controllers are each configured to manage a given rack in the plurality of racks comprising a given plurality of the rack-mounted computing devices; and

determining, with at least one of the first plurality of other rack controllers, that the first rack controller has failed;

electing, with at least one of the first plurality of other rack controllers, one of the first plurality of other rack controllers to serve as a replacement to the first rack controller in at least some roles filled by the first rack controller; and

distributing a result of the election to each the first plurality of other rack controllers.

24. The system of claim 23, wherein: (1) (1)

determining roles of rack controllers in a distributed peer-to-peer data-center management system comprises: selecting a primary rack controller; and

selecting a plurality of leader rack controllers by concurrently executing a plurality of respective instances of a distributed algorithm by which the respective leader is determined by consensus, each instance being executed by one of the second pluralities of other rack controllers to select a respective leader for the respective one of the second pluralities of other rack controllers.

25. The system of claim 24, wherein selecting a plurality of leader rack controllers by concurrently executing a plurality of respective instances of a distributed algorithm by which the respective leader is determined by consensus comprises: (0) (7)

detecting with a first one of the rack controllers in the one of the second pluralities of rack controllers that a signal has not been received from a leader within a threshold duration of time;

incrementing or decrementing a count with the with first one of the rack controllers;

sending, with the first one of the rack controllers, a request to vote for a new leader along with the incremented or decremented count to other rack controllers in the same second plurality of other rack controllers;

receiving the request with a second one of the rack controllers;

determining, with the second one of the rack controllers, to vote for the first one of the rack controllers, at least in part, by: determining that the second one of the rack controllers has not yet submitted a vote associated with the incremented or decremented count; and

determining that the request is the first request received by the second one of the rack controllers in a given round of voting;

sending, with the second one of the rack controllers, the vote to other rack controllers in the same second plurality of other rack controllers; and

selecting a leader based on amounts of votes received by candidate rack controllers.

26. The system of claim 23, wherein: (1) (1)

28. The system of claim 23, wherein: (0) (1)

29. The system of claim 23, effectuating operations comprising: (0) (1)

30. The system of claim 23, wherein: (0) (2)

31. The system of claim 23, wherein: (0) (1)

32. The system of claim 23, wherein: (0) (1)

34. The system of claim 23, wherein: (0) (1)

35. The system of claim 23, wherein: (0) (1)

36. The system of claim 23, wherein the command is one of: (0) (2)

37. The system of claim 23, wherein effectuating the operations implements control and updates of rack controllers at the rack-level, in a distributed, concurrent routine. (0)

38. The system of claim 23, wherein for each of the first plurality of other rack controllers, distributing, based at least some of the roles, via the out-of-band network, the data-center management command to a respective second plurality of other rack controllers comprises: (0) (2)

39. The system of claim 23, comprising: (0) (1)

40. The system of claim 23, comprising: (0) (1)

View Abstract and Specification Size

PDF with Images and Document Face >

Full Text Publication >

Patent US10257268

Distributed peer-to-peer data center management

Patent Matrix® Search