This thesis combines two statistical concepts: probabilistic graphical models and distributed algorithms. This type of `divide and conquer’ algorithm is becoming increasingly popular these days due to new challenges in collecting datasets, including security and privacy concerns, limited capacity of systems, and the presence of heterogeneous datasets from various locations. The broad scope of the thesis is to contribute to the estimation and inferential tools for Gaussian graphical models in a distributed setting. This setup presents a natural problem where datasets from different locations may have different sizes, resulting in an unbalanced distributed setting with different sub-sample sizes. In this context, a key question is: ``Can one devise an algorithm for estimation and inference in Gaussian graphical models under an unbalanced distributed setting that ensures the proposed aggregated solution remains consistent and asymptotically efficient, similar to the centralized procedure that would be obtained if all the data were available at a central location?’’ To address this question, we explore different scenarios. First, we consider a zero-mean Gaussian graphical model and develop the estimation and inferential tools within this family under the unbalanced distributed setting. Afterwards, we delve into the performance of the proposed procedure by investigating the false discovery rate under this setting. Consequently, we develop a procedure to control the false discovery rate for the hypothesis tests related to the entries of the precision matrix. Finally, to broaden the scope of our results, we extend our study to a non-zero Gaussian graphical model, where the mean of the model is affected by some covariates. We thoroughly investigate the estimation and inference procedures within this extended setting as well. Finally, we introduce an R package called ``DistributedGGL’’ to implement the procedures developed in this thesis in application.